12 Dec, 2023

How to translate manually uploaded machine data files

How to translate manually uploaded machine data files

About

In this tutorial, we will guide you through the steps of translating manually uploaded machine data files to retrieve planting, application, harvest and tillage operations in a consistent JSON format via Leaf. Manual machine file upload is useful for customers who want to submit their files via upload instead of 3rd party cloud provider API connections (e.g. John Deere Operation Center, Climate Fieldview, etc.).

Why use Leaf manual file upload?

  • Reach farmers who do not use cloud-based tools.
  • Access clean, merged and standardized data from various file formats.
  • Build faster using pre-built tools and infrastructure.

Summary


1. Register for a Leaf account

First, you’ll need to register for a Leaf account. If you don’t yet have an account you can register below. Make sure to activate your account via the activation email.

Register with Leaf -->

2. Authenticate to access Leaf’s API

Next, you’ll need to authenticate by requesting your access token:

import requests
url = "https://api.withleaf.io/api/authenticate"
data = {'username':'your email', 'password':'your password', 'rememberMe':'true'}
headers = {'Content-Type': 'application/json'}
response = requests.request("POST", url, headers=headers, json=data)
view raw authenticate.py hosted with ❤ by GitHub

View documentation -->

3. Create a Leaf user

Next, you’ll need to create a ‘Leaf user’.

What is a Leaf user? A Leaf user represents a data owner and is equivalent to a customer/grower account. A Leaf user helps keep your customers’ data organized under your API owner.

You can organize & filter data by leaf_user_id directly within Leaf and attach 3rd party API credentials to leaf_users to manage their individual account access.

Leaf, data infrastructure for agriculture

Your Leaf account includes a sample leaf_user with pre-loaded data for testing. Let's find that leaf_user & associated leaf_user_id by querying for all Leaf users in the account as shown below:

# Get all leaf_users in an account (with pagination)
def get_all_leaf_users(page_size=20):
all_users = []
page = 0
endpoint = 'https://api.withleaf.io/services/usermanagement/api/users'
headers = {'Authorization': f'Bearer {TOKEN}'}
while True:
params = {'page': page, 'size': page_size}
response = requests.get(endpoint, headers=headers, params=params)
if response.status_code != 200:
return None
all_users.extend(response.json())
if len(all_users) >= int(response.headers.get('X-Total-Count', 0)):
return all_users
page += 1
# Get the sample user and set leaf_user to the sample user
leaf_users = get_all_leaf_users()
for user in leaf_users:
if user['name'] == 'Sample User':
leaf_user_id = user['id']

In the previous step we set the leaf_user as the sample user. If you want to use a different user, you have the option to create a new Leaf user and set the user ID as shown below.

#Set the leaf_user_id
leaf_user_id = ''
view raw gistfile1.py hosted with ❤ by GitHub

4. Upload and convert a file

Now you can upload a file for Leaf to convert. Here are a few things to note:

  • The upload endpoint accepts .zip files and returns a batch_id.
  • If a nested .zip with multiple files is uploaded, Leaf will recursively unzip the files and attach a list with multiple file_ids to the batch ID.
  • If you don't know which kind of file(s) you have in your .zip, you can set the 'provider' parameter to 'Other' and Leaf will detect file type based on its extension.

You’ll find more detailed information about file upload endpoints, supported file types and request examples for cURL, Python and Javascript in the documentation here otherwise follow along with the example below.

import json
# Post a .zip to Leaf's upload endpoint
def batch_file_upload(leaf_user_id, file_to_convert):
endpoint = "https://api.withleaf.io/services/operations/api/batch"
files= {'file': file_to_convert}
params = {'leafUserId': leaf_user_id, 'provider': 'Other'}
headers = {'Authorization': f'Bearer {TOKEN}'}
response = requests.post(endpoint, headers=headers, files=files, params=params)
return response.json()
# prompt the upload
file_path = ""
file_to_convert = open(file_path, 'rb')
batch_upload_response = batch_file_upload(leaf_user_id, file_to_convert)
batch_id = batch_upload_response['id']
print (json.dumps(batch_upload_response, indent =2))

Fill in the file_path variable with the zip file path.

View ‘upload a file’ endpoint -->

5. Get the file IDs

Now that your file has been uploaded, you can use the batch_id to check if the process is finished ("PROCESSED") in the status property and get access to a list of the file_ids discovered and converted during the process ("leafFiles" properties).

# Get batch ID information
def get_batch_info(leaf_user_id, batch_id):
endpoint = f"https://api.withleaf.io/services/operations/api/batch/{batch_id}/"
params = {'leafUserId': leaf_user_id}
headers = {'Authorization': f'Bearer {TOKEN}'}
response = requests.get(endpoint, headers=headers, params=params)
return response.json()
batch_info = get_batch_info(leaf_user_id, batch_id)
print (json.dumps(batch_info, indent=2))
view raw get-batch.py hosted with ❤ by GitHub

The file_id will be used to get the converted file (Leaf’s standardGeojson format) in the next steps.

View ‘get batch upload’ endpoint -->

6. Get files

Now you can test the other machine file conversion endpoints to get a file's summary, status, units and images.

You can view sample machine file data here.

Below are examples of each request:

# Set the file ID
file_id = '{file_id}'
view raw file_id.py hosted with ❤ by GitHub

Summary

# Get converted file summary information
def get_file_summary(file_id):
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}'
headers = {'Authorization': f'Bearer {TOKEN}'}
response = requests.get(endpoint, headers=headers)
return (response.json())
# Print JSON response
print (json.dumps(get_file_summary(file_id), indent=2))
view raw summary.py hosted with ❤ by GitHub

Status

def get_file_status(file_id):
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}/status'
headers = {'Authorization': f'Bearer {TOKEN}'}
response = requests.get(endpoint, headers=headers)
return response.json()
print (json.dumps(get_file_status(file_id), indent=2))
view raw get_status.py hosted with ❤ by GitHub

Units

def get_file_units(file_id):
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}/units'
headers = {'Authorization': f'Bearer {TOKEN}'}
response = requests.get(endpoint, headers=headers)
return response.json()
print (json.dumps(get_file_units(file_id), indent=2))
view raw units.py hosted with ❤ by GitHub

Images

Please note that the below example requests images from machine files only, which tend to be fragmented across a field, so there will likely be gaps in the image. field operations are the recommended images to use as they're more comprehensive and have more cleaning/filtering options available.

def get_file_images(file_id):
endpoint = f'https://api.withleaf.io/services/operations/api/files/{file_id}/images'
headers = {'Authorization': f'Bearer {TOKEN}'}
response = requests.get(endpoint, headers=headers)
return response.json()
# Fetch image URLs based on the file ID
file_images = get_file_images(file_id)
view raw get-image.py hosted with ❤ by GitHub

7. Our last tips

You've now imported machine data into Leaf via manual file upload, great job!

We have a few last tips before you go:

  • Use the Leaf Link widget for manual file upload to save time on building UI; it is available in Angular and React. Leaf Magic Link is also available as a sharable URL so you don't have to build any UI and you use less code.
  • Set up your configurations and alerts prior to reading in data to suit your needs/preferences.
  • If you need help identifying issues with batch upload, the code below offers valuable info for troubleshooting, summarizing the total number of processed and failed files and the failure reasons for a given batch_id.
def fetch_file_status(file_id):
url = f"https://api.withleaf.io/services/operations/api/files/{file_id}/status"
return requests.get(url, headers={'Authorization': f'Bearer {TOKEN}'})
def process_file_status(file_status):
return {
step: value.get("message")
for step, value in file_status.items()
if value.get("status") == "failed" and value.get("message") != "skipped"
}
def summarize_failures(failure_conditions):
message_counts = Counter()
for failure_dict in failure_conditions:
for step, message in failure_dict.items():
message_counts[message] += 1
print("--- Failure Analysis ---")
for message, count in message_counts.items():
print(f"\n- Failure Condition:")
print(f" - Type: {message}")
print(f" - Count: {count}")
def get_summary_of_batch_file_status(leaf_user_id, batch_id):
batch_info = get_batch_info(leaf_user_id, batch_id) # Assuming this function is defined elsewhere
file_ids = batch_info.get("leafFiles", [])
success_count = 0
failure_count = 0
unique_failures = []
for file_id in file_ids:
try:
response = fetch_file_status(file_id)
response.raise_for_status()
failure_dict = process_file_status(response.json())
if failure_dict:
failure_count += 1
unique_failures.append(failure_dict)
else:
success_count += 1
except requests.RequestException as e:
print(f"API error for file ID {file_id}: {e}")
print(f"\n--- Summary ---\nTotal Success: {success_count}\nTotal Failures: {failure_count}")
summarize_failures(unique_failures)
get_summary_of_batch_file_status(leaf_user_id, batch_id)

That concludes this tutorial on how to get set up with manual file upload. To continue reading in and processing data, please refer to the 'machine file conversion' and 'field operation' parts of the documentation, which are the same as when you integrate a provider. Before you continue, be sure to read the machine file conversion overview to understand:

Leaf, data infrastructure for agriculture

FAQ: how do my customers get files onto a thumb drive in the first place?

--> If your grower customers are unsure how to export files from the monitor onto a thumb drive, this helpful resource from IA state guides you through it step-by-step depending on the machinery provider and the type of operation!

If you have any further questions, please don't hesitate to send us an email at help@withleaf.io.

Where to next?

--> Magic Link--> Field Boundaries --> Crop Monitoring --> Input Validator--> Magic Link --> Back to 'for developers'

Ready to begin?

Get a Demo and Start Building Today!