AWS IoT

Sending files to AWS S3 Bucket using Zephyr RTOS HTTP API

Jakub Zimnol
Nov 16, 2024

Introduction

An embedded device is typically designed to collect data over time — whether it’s logs, sensor readings, or even images captured by a small camera attached to the device. But how can a user retrieve this data after days, weeks, or even months of continuous operation? The most straightforward method is often through a USB connection, but what if the device is located hundreds of kilometers away, as is frequently the case with IoT devices?

One solution is for the device to send data chunks to the cloud periodically. However, this approach can quickly become chaotic to manage, especially if the data arrive out of order (thank you, UDP...). To streamline this, a more efficient method is to send a complete file over the air to a cloud-based storage server. But how can a small embedded device handle this reliably, especially when the entire file cannot fit into RAM to be sent as a single buffer payload?

It’s often said that UDP/CoAP is the best choice for low-power applications when sending small amounts of data. But what happens when the file is larger than 1 MB? In such cases, transmission speed becomes crucial — the less time the device spends sending the data, the less energy it will consume. This is where HTTP shines like a diamond, practically screaming "Use me!". Widely used by cloud servers, HTTP provides a reliable and efficient way to transfer large files from embedded devices to almost any server, making it an excellent choice for these use cases.

In this short tutorial, we’ll cover how to send a file from an embedded device running on Zephyr RTOS directly to an AWS S3 bucket. We’ll be using Zephyr RTOS with its generic HTTP API to handle the HTTP PUT operation, and to simplify the process, we will use a no-authentication method to upload the file to the S3 bucket.

Prerequisities

To follow the tutorial along, you’ll need the following:

  • A Zephyr RTOS SDK installed on your machine
  • A Zephyr RTOS-supported development board with any network connectivity (e.g. Wi-Fi or cellular)
  • An already created Zephyr RTOS application with any network connectivity
    • This tutorial will focus solely on the HTTP client operations needed to send data to AWS S3
  • Ten minutes of your time

Setting up the AWS S3 bucket

Amazon S3 (Simple Storage Service) is a cloud-based storage solution that makes it easy to store and manage large amounts of data. Files, also called "objects", are stored within "buckets", which act like containers for organizing and securing your data. S3 is highly scalable, meaning it can handle anything from a single file (like a favorite photo of your dog) to thousands of files, such as all your favorite photos of your dog log files generated by a fleet of IoT devices.

To create an S3 bucket, follow the first two steps from the Getting started with Amazon S3 offcial guide. After you create a bucket, save the bucket name and the region where the bucket is created in the <bucket_name>.s3.<region>.amazonaws.com format (e.g. my-example-bucket.s3.eu-north-1.amazonaws.com) - it will be required later in the tutorial.

After creating the bucket, you need to configure the bucket policy to allow performing the PUT operation on the bucket. To do so, go to Amazon S3 -> Buckets -> <bucket_name> -> Permissions -> Bucket policy and paste the following policy (replace <bucket_name> with your bucket name):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::<bucket_name>/*"
        }
    ]
}

After that, your bucket is ready to accept files via the HTTP PUT operation and you can proceed to the next step.

Zephyr RTOS application code step-by-step explanation

In this tutorial, we’ll focus on sending a file to an AWS S3 bucket. For demonstration purposes, we’ll use a simple snippet of Lorem Ipsum text, but in a real-world scenario, this could be replaced with actual file content read from non-volatile memory, such as an SD card.

So let's dive into the code!

Socket operations

The socket_connect() function establishes a socket connection to a specified server and port. It uses the POSIX API — specifically, the getaddrinfo() function — to obtain the server's address, then creates a socket based on this address information. Once the socket is created, it attempts to connect to the server. This function is utilized within send_file_via_http() to handle the connection setup.

The socket_cleanup() function handles resource cleanup, freeing the socket and address information once the connection is no longer needed.

void socket_cleanup(int sock, struct addrinfo *addr)
{
        close(sock);
        freeaddrinfo(addr);
}

static int setup_socket(sa_family_t family, const char *server, const char *port_str, int *sock,
                        struct addrinfo **addr, socklen_t addr_len)
{
        struct addrinfo hints;
        int ret;

        memset(&hints, 0, sizeof(hints));
        hints.ai_family = AF_INET;
        hints.ai_socktype = SOCK_STREAM;

        ret = getaddrinfo(server, port_str, &hints, addr);
        if (ret < 0) {
                LOG_ERR("getaddrinfo error: %d", ret);
                return -1;
        }

        *sock = socket((*addr)->ai_family, (*addr)->ai_socktype, (*addr)->ai_protocol);

        if (*sock < 0) {
                LOG_ERR("Failed to create IPv4 HTTP socket (%d)", -errno);
                freeaddrinfo(*addr);
                return -1;
        }

        return 0;
}

int socket_connect(sa_family_t family, const char *server, const char *port, int *sock,
                   struct addrinfo **addr, socklen_t addr_len)
{
        int ret;

        ret = setup_socket(family, server, port, sock, addr, addr_len);
        if (ret < 0 || *sock < 0) {
                return -1;
        }

        struct sockaddr_in *ipv4_addr = (struct sockaddr_in *)(*addr)->ai_addr;
        ret = connect(*sock, (struct sockaddr *)ipv4_addr, sizeof(*ipv4_addr));
        if (ret < 0) {
                LOG_ERR("Cannot connect to IPv4 remote (%d)", -errno);
                socket_cleanup(*sock, *addr);
                *sock = -1;
                ret = -errno;
        }

        return ret;
}

File content operations

The file_content_get_size() function returns the size of the file content to be sent. In this example, it returns the length of the LOREM_IPSUM text, but it can be modified to return the actual file size read from e.g. the file system.

The file_content_get_chunk() function returns a specific chunk of the file content based on the given offset. It’s used within the http_payload_cb() function to read and send the file content chunk by chunk over the HTTP connection, simulating file content retrieval from a source such as the file system.

An argument specifying the file path can be added to both functions, which is especially useful when reading file content from an external file system (such as an SD card). This allows the functions to retrieve file content directly from the specified path.

/* the LOREM_IPSUM string is not included in this code snippet,
   see the full file_content.c file below */

size_t file_content_get_size(void)
{
        return strlen(LOREM_IPSUM);
}

size_t file_content_get_chunk(char *out_buffer, size_t max_size, size_t offset)
{
        size_t size = file_content_get_size();
        if (offset >= size) {
                return 0;
        }

        size_t chunk_size = MIN(size - offset, max_size);

        memcpy(out_buffer, LOREM_IPSUM + offset, chunk_size);

        return chunk_size;
}

HTTP request operations

The send_file_via_http() function is the main function responsible for sending a file over HTTP to the AWS S3 bucket. It first connects to the S3 bucket using the socket_connect() function, then prepares the HTTP request headers, including the Content-Length header with the file size. Next, it configures the HTTP request with the necessary information, such as the HTTP method, URL, host, payload callback, and response callback. The request is then sent using the http_client_req() function, and once completed, the socket connection is cleaned up with the socket_cleanup() function. Note that this function should be called only after the network connection (e.g. Wi-Fi or cellular) is established.

The http_response_cb() function is a callback that executes upon receiving an HTTP response. It logs the response status and any data received. The expected server response is OK, which corresponds to a status code of 200.

The http_payload_cb() function is a callback triggered when the HTTP client needs to send the entire payload data. It reads the file content in chunks using the file_content_get_chunk() function and transmits it with the send() function. It’s crucial that http_payload_cb() sends (and returns) exactly the same amount of data specified by the Content-Length header in the send_file_via_http() function.

The HTTP_PORT macro defines the port used for the HTTP connection set to the default HTTP port 80. The S3_BUCKET_ADDRESS macro specifies the address of your S3 bucket and should be replaced with your actual bucket address. The HTTP_REQUEST_URL macro defines the URL to which the file will be sent, which, in other words, is the path where the file will be stored in the bucket. Note that the specified path does not have to pre-exist in the bucket; it will be created automatically upon file upload.

#define HTTP_PORT         "80"
#define S3_BUCKET_ADDRESS "my-example-bucket.s3.eu-north-1.amazonaws.com" // replace with your bucket address
#define HTTP_REQUEST_URL  "/zephyr_files/lorem_ipsum.txt"

static void http_response_cb(struct http_response *rsp, enum http_final_call final_data,
                                 void *user_data)
{
        if (final_data == HTTP_DATA_MORE) {
                LOG_INF("Partial data received (%zd bytes)", rsp->data_len);
        } else if (final_data == HTTP_DATA_FINAL) {
                LOG_INF("All the data received (%zd bytes)", rsp->data_len);
        }

        LOG_INF("Response to: %s, status: %s", (const char *)user_data, rsp->http_status);
}

static int http_payload_cb(int sock, struct http_request *req, void *user_data)
{
        int ret = 0;
        size_t offset = 0;
        char read_buf[1024];
        memset(read_buf, 0, sizeof(read_buf));

        while (true) {
                ssize_t read = file_content_get_chunk(read_buf, sizeof(read_buf), offset);
                if (read < 0) {
                        LOG_ERR("Failed to get file chunk");
                        return -1;
                } else if (read == 0) {
                        LOG_INF("Finished reading file");
                        break;
                }

                offset += read;
                ssize_t sent = send(sock, read_buf, read, 0);
                if (sent < 0) {
                        LOG_ERR("Send error, abort");
                        return -1;
                }
                LOG_INF("Sent %d bytes", sent);
                ret += sent;
        }

        LOG_INF("HTTP payload sent: %d", ret);

        return ret;
}

int send_file_via_http(void)
{
        uint8_t http_recv_buf[512];
        struct addrinfo *addr;
        int ret;
        int sock = -1;

        size_t file_size = file_content_get_size();
        if (file_size == 0) {
                LOG_WRN("File size == 0, skip");
                return 0;
        }

        ret = socket_connect(AF_INET, S3_BUCKET_ADDRESS, HTTP_PORT, &sock, &addr, sizeof(*addr));
        if (ret) {
                return -1;
        }

        char content_len_header[64];
        memset(content_len_header, 0, sizeof(content_len_header));
        snprintf(content_len_header, sizeof(content_len_header), "Content-Length: %zd\r\n",
                 file_size);
        const char *http_headers[] = {content_len_header, NULL};

        LOG_INF("HTTP Header: %s", content_len_header);
        LOG_INF("Request url: %s", HTTP_REQUEST_URL);

        struct http_request req;
        memset(&req, 0, sizeof(req));
        req.method = HTTP_PUT;
        req.protocol = "HTTP/1.1";
        req.url = HTTP_REQUEST_URL;
        req.host = S3_BUCKET_ADDRESS;
        req.payload_cb = http_payload_cb;
        req.header_fields = http_headers;
        req.response = http_response_cb;
        req.recv_buf = http_recv_buf;
        req.recv_buf_len = sizeof(http_recv_buf);

        ret = http_client_req(sock, &req, 3 * 1000, "IPv4 PUT");

        socket_cleanup(sock, addr);

        return ret;
}

General project structure

To simplify the understanding and possible later integration with your application, the code can be organized into several files:

  1. sockets.h and sockets.c - These files contain the basic socket operations required for making HTTP requests.
  2. file_content.h and file_content.c - These files contain the content of the file (and operations related to it) that will be sent to the AWS S3 bucket. As noted earlier, this can be modified to retrieve actual file content from non-volatile memory.
  3. file_sender.h and file_sender.c - These files contain the core logic for sending files.
  4. prj.conf - This file includes the Zephyr configuration settings required to compile and run the project.

The files can be organized as follows:

sockets.h

#pragma once

#include <zephyr/net/socket.h>

int socket_connect(sa_family_t family, const char *server, const char *port, int *sock,
                   struct addrinfo **addr, socklen_t addr_len);
void socket_cleanup(int sock, struct addrinfo *addr);

sockets.c

#include <zephyr/kernel.h>
#include <zephyr/logging/log.h>

#include <zephyr/net/net_ip.h>
#include <zephyr/net/socket.h>

#include "file_sender.h"
#include "sockets.h"

LOG_MODULE_REGISTER(sockets, LOG_LEVEL_INF);

void socket_cleanup(int sock, struct addrinfo *addr)
{
        close(sock);
        freeaddrinfo(addr);
}

static int setup_socket(sa_family_t family, const char *server, const char *port_str, int *sock,
                        struct addrinfo **addr, socklen_t addr_len)
{
        struct addrinfo hints;
        int ret;

        memset(&hints, 0, sizeof(hints));
        hints.ai_family = AF_INET;
        hints.ai_socktype = SOCK_STREAM;

        ret = getaddrinfo(server, port_str, &hints, addr);
        if (ret < 0) {
                LOG_ERR("getaddrinfo error: %d", ret);
                return -1;
        }

        *sock = socket((*addr)->ai_family, (*addr)->ai_socktype, (*addr)->ai_protocol);

        if (*sock < 0) {
                LOG_ERR("Failed to create IPv4 HTTP socket (%d)", -errno);
                freeaddrinfo(*addr);
                return -1;
        }

        return 0;
}

int socket_connect(sa_family_t family, const char *server, const char *port, int *sock,
                   struct addrinfo **addr, socklen_t addr_len)
{
        int ret;

        ret = setup_socket(family, server, port, sock, addr, addr_len);
        if (ret < 0 || *sock < 0) {
                return -1;
        }

        struct sockaddr_in *ipv4_addr = (struct sockaddr_in *)(*addr)->ai_addr;
        ret = connect(*sock, (struct sockaddr *)ipv4_addr, sizeof(*ipv4_addr));
        if (ret < 0) {
                LOG_ERR("Cannot connect to IPv4 remote (%d)", -errno);
                socket_cleanup(*sock, *addr);
                *sock = -1;
                ret = -errno;
        }

        return ret;
}

file_content.h

#pragma once

#include <zephyr/types.h>

size_t file_content_get_size(void);
size_t file_content_get_chunk(char *out_buffer, size_t max_size, size_t offset);

file_content.c

#include <zephyr/sys/util.h>

#include <string.h>

#include "file_content.h"

static const char *LOREM_IPSUM =
        "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean eget orci venenatis, "
        "volutpat lorem at, "
        "porttitor mauris. Sed eget urna varius, accumsan libero at, sagittis dui. Maecenas "
        "malesuada, orci non interdum "
        "fringilla, quam ligula dictum metus, et convallis odio lorem vitae dui. Duis vulputate "
        "metus eget suscipit "
        "fermentum. Donec sit amet eros at elit egestas convallis. Cras tincidunt scelerisque "
        "dolor, et tincidunt elit "
        "tempor sit amet. Nam id arcu non est suscipit fermentum id in arcu. Curabitur vitae lorem "
        "ut nisi eleifend "
        "tincidunt.\n"
        "\n"
        "Nullam eget fermentum nunc. Integer vitae nisi in nulla interdum fermentum at et lectus. "
        "Morbi ultrices, "
        "nisi at convallis bibendum, odio urna sollicitudin risus, nec ullamcorper nunc lectus non "
        "felis. Proin auctor "
        "lacus eget mauris gravida, nec malesuada lorem pharetra. Sed nec ex non mauris efficitur "
        "fringilla. "
        "Pellentesque pellentesque fermentum turpis, ut gravida nulla commodo in. Suspendisse "
        "potenti. Fusce vitae "
        "quam ut libero tristique malesuada non et ligula. Vivamus in ante at purus posuere "
        "accumsan vitae at purus.\n"
        "\n"
        "Curabitur malesuada sapien sit amet arcu volutpat, vel laoreet risus malesuada. Nulla "
        "facilisi. Phasellus "
        "nec felis vel nulla feugiat varius. Aliquam erat volutpat. Integer at facilisis ligula. "
        "Fusce id sapien ligula. "
        "Maecenas maximus, eros eget facilisis pharetra, magna magna elementum nulla, in commodo "
        "justo sapien id risus. "
        "Sed vestibulum elit ac felis tincidunt varius.\n"
        "\n"
        "Quisque accumsan sapien magna, sit amet fermentum nisl eleifend sit amet. Duis "
        "sollicitudin, nisi in porta "
        "euismod, lectus mi pellentesque quam, ut bibendum ex nisl et arcu. Ut fermentum, libero "
        "sed gravida interdum, "
        "ligula odio facilisis mauris, vitae fringilla lectus odio nec libero. Sed a velit eget "
        "arcu aliquet congue.\n"
        "\n"
        "Donec luctus eros sit amet augue accumsan, at convallis erat egestas. Etiam finibus, "
        "dolor sed cursus laoreet, "
        "est justo laoreet elit, at vehicula est libero id justo. Mauris elementum erat risus, a "
        "dignissim urna elementum "
        "a. Nullam pulvinar, tortor sed sollicitudin dignissim, metus orci scelerisque eros, et "
        "dictum sapien ex id leo. "
        "Aliquam varius, velit a commodo bibendum, urna mauris porttitor leo, nec congue orci arcu "
        "sit amet ligula. Nam "
        "egestas nibh sed nisl pellentesque, id sagittis ligula cursus. Sed eget diam auctor orci "
        "suscipit hendrerit. "
        "Cras at lectus sed urna facilisis feugiat. Duis porta turpis odio, non auctor nisi "
        "efficitur non. Nulla "
        "facilisi.\n";

size_t file_content_get_size(void)
{
        return strlen(LOREM_IPSUM);
}

size_t file_content_get_chunk(char *out_buffer, size_t max_size, size_t offset)
{
        size_t size = file_content_get_size();
        if (offset >= size) {
                return 0;
        }

        size_t chunk_size = MIN(size - offset, max_size);

        memcpy(out_buffer, LOREM_IPSUM + offset, chunk_size);

        return chunk_size;
}

file_sender.h

#pragma once

int send_file_via_http(void);

file_sender.c

#include <zephyr/logging/log.h>
#include <zephyr/kernel.h>
#include <zephyr/net/socket.h>
#include <zephyr/net/http/client.h>

#include "file_content.h"
#include "file_sender.h"
#include "sockets.h"

LOG_MODULE_REGISTER(file_sender, LOG_LEVEL_INF);

#define HTTP_PORT         "80"
#define S3_BUCKET_ADDRESS "my-example-bucket.s3.eu-north-1.amazonaws.com" // replace with your bucket address
#define HTTP_REQUEST_URL  "/zephyr_files/lorem_ipsum.txt"

static void http_response_cb(struct http_response *rsp, enum http_final_call final_data,
                             void *user_data)
{
        if (final_data == HTTP_DATA_MORE) {
                LOG_INF("Partial data received (%zd bytes)", rsp->data_len);
        } else if (final_data == HTTP_DATA_FINAL) {
                LOG_INF("All the data received (%zd bytes)", rsp->data_len);
        }

        LOG_INF("Response to: %s, status: %s", (const char *)user_data, rsp->http_status);
}

static int http_payload_cb(int sock, struct http_request *req, void *user_data)
{
        int ret = 0;
        size_t offset = 0;
        char read_buf[1024];
        memset(read_buf, 0, sizeof(read_buf));

        while (true) {
                ssize_t read = file_content_get_chunk(read_buf, sizeof(read_buf), offset);
                if (read < 0) {
                        LOG_ERR("Failed to get file chunk");
                        return -1;
                } else if (read == 0) {
                        LOG_INF("Finished reading file");
                        break;
                }

                offset += read;
                ssize_t sent = send(sock, read_buf, read, 0);
                if (sent < 0) {
                        LOG_ERR("Send error, abort");
                        return -1;
                }
                LOG_INF("Sent %d bytes", sent);
                ret += sent;
        }

        LOG_INF("HTTP payload sent: %d", ret);

        return ret;
}

int send_file_via_http(void)
{
        uint8_t http_recv_buf[512];
        struct addrinfo *addr;
        int ret;
        int sock = -1;

        size_t file_size = file_content_get_size();
        if (file_size == 0) {
                LOG_WRN("File size == 0, skip");
                return 0;
        }

        ret = socket_connect(AF_INET, S3_BUCKET_ADDRESS, HTTP_PORT, &sock, &addr, sizeof(*addr));
        if (ret) {
                return -1;
        }

        char content_len_header[64];
        memset(content_len_header, 0, sizeof(content_len_header));
        snprintf(content_len_header, sizeof(content_len_header), "Content-Length: %zd\r\n",
                 file_size);
        const char *http_headers[] = {content_len_header, NULL};

        LOG_INF("HTTP Header: %s", content_len_header);
        LOG_INF("Request url: %s", HTTP_REQUEST_URL);

        struct http_request req;
        memset(&req, 0, sizeof(req));
        req.method = HTTP_PUT;
        req.protocol = "HTTP/1.1";
        req.url = HTTP_REQUEST_URL;
        req.host = S3_BUCKET_ADDRESS;
        req.payload_cb = http_payload_cb;
        req.header_fields = http_headers;
        req.response = http_response_cb;
        req.recv_buf = http_recv_buf;
        req.recv_buf_len = sizeof(http_recv_buf);

        ret = http_client_req(sock, &req, 3 * 1000, "IPv4 PUT");

        socket_cleanup(sock, addr);

        return ret;
}

prj.conf

# Logging
CONFIG_LOG=y

# Network
CONFIG_NETWORKING=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_IPV4=y
CONFIG_NET_CONFIG_NEED_IPV4=y

# HTTP requests
CONFIG_HTTP_CLIENT=y

# rest of the configuration

 

Building and running the project

To send files to the AWS S3 bucket, simply copy the .c and .h files listed above into your Zephyr RTOS project and include them in the build (for example, by adding them to the SOURCES list in the CMakeLists.txt file). After that, you can call the send_file_via_http() function in your application to send the file to the bucket. Be sure to:

  1. Replace the S3_BUCKET_ADDRESS macro value in the file_sender.c file with your bucket address.
  2. Call the send_file_via_http() function only after the network connection is established.
  3. Set up your AWS S3 bucket and configure the bucket policy to allow the PUT operation for file uploads.

After flashing the board, open the serial terminal to view the logs. Example logs should look like this:

...
[00:00:03.509,063] <inf> file_sender: Connected to network, sending file
[00:00:04.045,898] <inf> file_sender: HTTP Header: Content-Length: 2455
[00:00:04.045,928] <inf> file_sender: Request url: /zephyr_files/lorem_ipsum.txt
[00:00:04.046,997] <inf> file_sender: Sent 1024 bytes
[00:00:04.047,607] <inf> file_sender: Sent 1024 bytes
[00:00:04.048,034] <inf> file_sender: Sent 407 bytes
[00:00:04.048,065] <inf> file_sender: Finished reading file
[00:00:04.048,065] <inf> file_sender: HTTP payload sent: 2455
[00:00:04.340,301] <inf> file_sender: All the data received (319 bytes)
[00:00:04.340,393] <inf> file_sender: Response to: IPv4 PUT, status: OK
...

If you see the Response to: IPv4 PUT, status: OK log message, it means that the file was successfully sent to the AWS S3 bucket. You can verify this by checking the content of the bucket in the AWS S3 console.

file_in_bucket

Congratulations! You've successfully sent a file from your Zephyr RTOS device to an AWS S3 bucket using the HTTP PUT operation.

Summary

In this tutorial, we’ve shown how to send a file from an embedded device running Zephyr RTOS directly to an AWS S3 bucket. This method can be applied to send log files, sensor data, pictures of your dog, or any other data from your device to the cloud for further analysis and processing.
AWS IoT Zephyr RTOS

Similar posts

Get notified on new IoT insights

Subscribe