About
In this tutorial, we will guide you through the steps of translating manually uploaded machine data files to retrieve planting, application, harvest and tillage operations in a consistent JSON format via Leaf. Manual machine file upload is useful for customers who want to submit their files via upload instead of 3rd party cloud provider API connections (e.g. John Deere Operation Center, Climate Fieldview, etc.).
Why use Leaf manual file upload?
- Reach farmers who do not use cloud-based tools.
- Access clean, merged and standardized data from various file formats.
- Build faster using pre-built tools and infrastructure.
Summary
1. Register for a Leaf account
First, you’ll need to register for a Leaf account. If you don’t yet have an account you can register below. Make sure to activate your account via the activation email.
2. Authenticate to access Leaf’s API
Next, you’ll need to authenticate by requesting your access token:
import requests | |
url = "https://api.withleaf.io/api/authenticate" | |
data = {'username':'your email', 'password':'your password', 'rememberMe':'true'} | |
headers = {'Content-Type': 'application/json'} | |
response = requests.request("POST", url, headers=headers, json=data) |
3. Create a Leaf user
Next, you’ll need to create a ‘Leaf user’.
What is a Leaf user? A Leaf user represents a data owner and is equivalent to a customer/grower account. A Leaf user helps keep your customers’ data organized under your API owner.
You can organize & filter data by leaf_user_id directly within Leaf and attach 3rd party API credentials to leaf_users to manage their individual account access.
Your Leaf account includes a sample leaf_user with pre-loaded data for testing. Let's find that leaf_user & associated leaf_user_id by querying for all Leaf users in the account as shown below:
# Get all leaf_users in an account (with pagination) | |
def get_all_leaf_users(page_size=20): | |
all_users = [] | |
page = 0 | |
endpoint = 'https://api.withleaf.io/services/usermanagement/api/users' | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
while True: | |
params = {'page': page, 'size': page_size} | |
response = requests.get(endpoint, headers=headers, params=params) | |
if response.status_code != 200: | |
return None | |
all_users.extend(response.json()) | |
if len(all_users) >= int(response.headers.get('X-Total-Count', 0)): | |
return all_users | |
page += 1 | |
# Get the sample user and set leaf_user to the sample user | |
leaf_users = get_all_leaf_users() | |
for user in leaf_users: | |
if user['name'] == 'Sample User': | |
leaf_user_id = user['id'] |
In the previous step we set the leaf_user as the sample user. If you want to use a different user, you have the option to create a new Leaf user and set the user ID as shown below.
#Set the leaf_user_id | |
leaf_user_id = '' |
4. Upload and convert a file
Now you can upload a file for Leaf to convert. Here are a few things to note:
- The upload endpoint accepts .zip files and returns a batch_id.
- If a nested .zip with multiple files is uploaded, Leaf will recursively unzip the files and attach a list with multiple file_ids to the batch ID.
- If you don't know which kind of file(s) you have in your .zip, you can set the 'provider' parameter to 'Other' and Leaf will detect file type based on its extension.
You’ll find more detailed information about file upload endpoints, supported file types and request examples for cURL, Python and Javascript in the documentation here otherwise follow along with the example below.
import json | |
# Post a .zip to Leaf's upload endpoint | |
def batch_file_upload(leaf_user_id, file_to_convert): | |
endpoint = "https://api.withleaf.io/services/operations/api/batch" | |
files= {'file': file_to_convert} | |
params = {'leafUserId': leaf_user_id, 'provider': 'Other'} | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
response = requests.post(endpoint, headers=headers, files=files, params=params) | |
return response.json() | |
# prompt the upload | |
file_path = "" | |
file_to_convert = open(file_path, 'rb') | |
batch_upload_response = batch_file_upload(leaf_user_id, file_to_convert) | |
batch_id = batch_upload_response['id'] | |
print (json.dumps(batch_upload_response, indent =2)) |
Fill in the file_path variable with the zip file path.
View ‘upload a file’ endpoint -->
5. Get the file IDs
Now that your file has been uploaded, you can use the batch_id to check if the process is finished ("PROCESSED") in the status property and get access to a list of the file_ids discovered and converted during the process ("leafFiles" properties).
# Get batch ID information | |
def get_batch_info(leaf_user_id, batch_id): | |
endpoint = f"https://api.withleaf.io/services/operations/api/batch/{batch_id}/" | |
params = {'leafUserId': leaf_user_id} | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
response = requests.get(endpoint, headers=headers, params=params) | |
return response.json() | |
batch_info = get_batch_info(leaf_user_id, batch_id) | |
print (json.dumps(batch_info, indent=2)) |
The file_id will be used to get the converted file (Leaf’s standardGeojson format) in the next steps.
View ‘get batch upload’ endpoint -->
6. Get files
Now you can test the other machine file conversion endpoints to get a file's summary, status, units and images.
You can view sample machine file data here.
Below are examples of each request:
# Set the file ID | |
file_id = '{file_id}' |
Summary
# Get converted file summary information | |
def get_file_summary(file_id): | |
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}' | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
response = requests.get(endpoint, headers=headers) | |
return (response.json()) | |
# Print JSON response | |
print (json.dumps(get_file_summary(file_id), indent=2)) |
Status
def get_file_status(file_id): | |
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}/status' | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
response = requests.get(endpoint, headers=headers) | |
return response.json() | |
print (json.dumps(get_file_status(file_id), indent=2)) |
Units
def get_file_units(file_id): | |
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}/units' | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
response = requests.get(endpoint, headers=headers) | |
return response.json() | |
print (json.dumps(get_file_units(file_id), indent=2)) |
Images
Please note that the below example requests images from machine files only, which tend to be fragmented across a field, so there will likely be gaps in the image. field operations are the recommended images to use as they're more comprehensive and have more cleaning/filtering options available.
def get_file_images(file_id): | |
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}/images' | |
headers = {'Authorization': f'Bearer {TOKEN}'} | |
response = requests.get(endpoint, headers=headers) | |
return response.json() | |
# Fetch image URLs based on the file ID | |
file_images = get_file_images(file_id) |
7. Our last tips
You've now imported machine data into Leaf via manual file upload, great job!
We have a few last tips before you go:
- Use the Leaf Link widget for manual file upload to save time on building UI; it is available in Angular and React. Leaf Magic Link is also available as a sharable URL so you don't have to build any UI and you use less code.
- Set up your configurations and alerts prior to reading in data to suit your needs/preferences.
- If you need help identifying issues with batch upload, the code below offers valuable info for troubleshooting, summarizing the total number of processed and failed files and the failure reasons for a given batch_id.
def fetch_file_status(file_id): | |
url = f"https://api.withleaf.io/services/operations/api/files/{file_id}/status" | |
return requests.get(url, headers={'Authorization': f'Bearer {TOKEN}'}) | |
def process_file_status(file_status): | |
return { | |
step: value.get("message") | |
for step, value in file_status.items() | |
if value.get("status") == "failed" and value.get("message") != "skipped" | |
} | |
def summarize_failures(failure_conditions): | |
message_counts = Counter() | |
for failure_dict in failure_conditions: | |
for step, message in failure_dict.items(): | |
message_counts[message] += 1 | |
print("--- Failure Analysis ---") | |
for message, count in message_counts.items(): | |
print(f"\n- Failure Condition:") | |
print(f" - Type: {message}") | |
print(f" - Count: {count}") | |
def get_summary_of_batch_file_status(leaf_user_id, batch_id): | |
batch_info = get_batch_info(leaf_user_id, batch_id) # Assuming this function is defined elsewhere | |
file_ids = batch_info.get("leafFiles", []) | |
success_count = 0 | |
failure_count = 0 | |
unique_failures = [] | |
for file_id in file_ids: | |
try: | |
response = fetch_file_status(file_id) | |
response.raise_for_status() | |
failure_dict = process_file_status(response.json()) | |
if failure_dict: | |
failure_count += 1 | |
unique_failures.append(failure_dict) | |
else: | |
success_count += 1 | |
except requests.RequestException as e: | |
print(f"API error for file ID {file_id}: {e}") | |
print(f"\n--- Summary ---\nTotal Success: {success_count}\nTotal Failures: {failure_count}") | |
summarize_failures(unique_failures) | |
get_summary_of_batch_file_status(leaf_user_id, batch_id) |
That concludes this tutorial on how to get set up with manual file upload. To continue reading in and processing data, please refer to the 'machine file conversion' and 'field operation' parts of the documentation, which are the same as when you integrate a provider. Before you continue, be sure to read the machine file conversion overview to understand:
- How the machine file conversion process works (raw files vs standardGeojsons vs operations and operation summaries)
- How and why Leaf's infrastructure merges files (saving you time building it yourself)
FAQ: how do my customers get files onto a thumb drive in the first place?
--> If your grower customers are unsure how to export files from the monitor onto a thumb drive, this helpful resource from IA state guides you through it step-by-step depending on the machinery provider and the type of operation!
If you have any further questions, please don't hesitate to send us an email at help@withleaf.io.
Where to next?
--> Magic Link--> Field Boundaries --> Crop Monitoring --> Input Validator--> Magic Link --> Back to 'for developers'