URL API
The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.
The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.
The library has configurable request/response timeouts and retries.
Features:
- Supports transfer-chunk encoding for dynamic request bodies.
- Control over headers
- Full-duplex streaming support for minimal memory usage.
- Full support for TLS including certificate verification.
- Configurable retries and timeouts.
- Parallelism via fiber coroutines.
The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.
Extensions
Url | Url request object. |
Functions
Url* | urlAlloc(void) |
Allocate a URL object. | |
void | urlClose(Url *up) |
Close the underlying network socket associated with the URL object. | |
int | urlFetch(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...) |
Fetch a URL. | |
Json* | urlFetchJson(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...) |
Fetch a URL and return a JSON response. | |
void | urlFree(Url *up) |
Free a URL object. | |
char* | urlGet(cchar *uri, cchar *headers, ...) |
Get a URL using a HTTP GET request. | |
cchar* | urlGetError(Url *up) |
Get the URL internal error message. | |
cchar* | urlGetHeader(Url *up, cchar *header) |
Get a response HTTP header. | |
Json* | urlGetJsonResponse(Url *up) |
Get the response to a URL request as a JSON object tree. | |
cchar* | urlGetResponse(Url *up) |
Get the response to a URL request. | |
int | urlGetStatus(Url *up) |
Get the HTTP response status code for a request. | |
int | urlParse(Url *up, cchar *uri) |
Parse a URL into its constituent components in the Url structure. | |
char* | urlPost(cchar *uri, cvoid *data, ssize size, cchar *headers, ...) |
Issue a HTTP POST request. | |
Json* | urlPostJson(cchar *uri, cvoid *data, ssize len, cchar *headers, ...) |
Issue a HTTP POST request and return parsed JSON. | |
ssize | urlRead(Url *up, char *buf, ssize bufsize) |
Read response data for a request. | |
void | urlSetCerts(Url *up, cchar *key, cchar *cert, cchar *ca, cchar *revoke) |
Define the certificates to use with TLS. | |
void | urlSetDefaultRetries(int retries) |
Set the default number of retries for requests. | |
void | urlSetDefaultTimeout(int timeout) |
Set the default request timeout to use for future URL instances. | |
void | urlSetError(Url *up, cchar *message, ...) |
Set the URL internal error message. | |
void | urlSetTimeout(Url *up, int timeout) |
Set the request timeout to use for the specific URL object. | |
void | urlSetVerify(Url *up, int verifyPeer, int verifyIssuer) |
Control verification of TLS connections. | |
int | urlStart(Url *up, cchar *method, cchar *uri, ssize txLen) |
Start a Url request. | |
ssize | urlWrite(Url *up, cvoid *data, ssize size) |
Write body data for a request. | |
int | urlWriteHeaders(Url *up, cchar *headers, ...) |
Write request headers. |
Typedefs
UrlProc | URL callback procedure. |
Defines
#define | URL_CODE_ACCEPTED 202 |
The request has been accepted and processing is continuing. | |
#define | URL_CODE_BAD_GATEWAY 502 |
The server cannot act as a gateway for the given request. | |
#define | URL_CODE_BAD_METHOD 405 |
The request HTTP method was not supported by the resource. | |
#define | URL_CODE_BAD_REQUEST 400 |
The request is malformed. | |
#define | URL_CODE_BAD_VERSION 505 |
The server does not support the HTTP protocol version. | |
#define | URL_CODE_CONFLICT 409 |
The request had a conflict in the request headers and URI. | |
#define | URL_CODE_CONTINUE 100 |
Continue with request, only parial content transmitted. | |
#define | URL_CODE_CREATED 201 |
The request has completed and a new resource was created. | |
#define | URL_CODE_EXPECTATION_FAILED 417 |
The server cannot satisfy the Expect header requirements. | |
#define | URL_CODE_FORBIDDEN 403 |
The request was legal, but the server refuses to process. | |
#define | URL_CODE_GATEWAY_TIMEOUT 504 |
The server gateway timed out waiting for the upstream server. | |
#define | URL_CODE_GONE 410 |
The requested resource is no longer available. | |
#define | URL_CODE_IM_A_TEAPOT 418 |
Short and stout error code (RFC 2324). | |
#define | URL_CODE_INSUFFICIENT_STORAGE 507 |
The server has insufficient storage to complete the request. | |
#define | URL_CODE_INTERNAL_SERVER_ERROR 500 |
Server processing or configuration error. | |
#define | URL_CODE_LENGTH_REQUIRED 411 |
The request did not specify a required content length. | |
#define | URL_CODE_MOVED_PERMANENTLY 301 |
The requested URI has moved permanently to a new location. | |
#define | URL_CODE_MOVED_TEMPORARILY 302 |
The URI has moved temporarily to a new location. | |
#define | URL_CODE_NO_CONTENT 204 |
The request has completed and there is no response to send. | |
#define | URL_CODE_NO_RESPONSE 444 |
The connection was closed with no response to the client. | |
#define | URL_CODE_NOT_ACCEPTABLE 406 |
The requested resource cannot generate the required content. | |
#define | URL_CODE_NOT_AUTHORITATIVE 203 |
The request has completed but content may be from another source. | |
#define | URL_CODE_NOT_FOUND 404 |
The requested resource was not found. | |
#define | URL_CODE_NOT_IMPLEMENTED 501 |
The server does not recognize the request or method. | |
#define | URL_CODE_NOT_MODIFIED 304 |
The requested resource has changed since the last request. | |
#define | URL_CODE_OK 200 |
The request completed successfully. | |
#define | URL_CODE_PARTIAL 206 |
The request has completed and is returning parial content. | |
#define | URL_CODE_PAYMENT_REQUIRED 402 |
Reserved for future use. | |
#define | URL_CODE_PRECOND_FAILED 412 |
The server cannot satisfy one of the request preconditions. | |
#define | URL_CODE_RANGE_NOT_SATISFIABLE 416 |
The request content range does not exist for the resource. | |
#define | URL_CODE_REQUEST_TIMEOUT 408 |
The server timed out waiting for the request to complete. | |
#define | URL_CODE_REQUEST_TOO_LARGE 413 |
The request is too large for the server to process. | |
#define | URL_CODE_REQUEST_URL_TOO_LARGE 414 |
The request URI is too long for the server to process. | |
#define | URL_CODE_RESET 205 |
The request has completed with no content. | |
#define | URL_CODE_SEE_OTHER 303 |
The requested URI can be found at another URI location. | |
#define | URL_CODE_SERVICE_UNAVAILABLE 503 |
The server is currently unavailable or overloaded. | |
#define | URL_CODE_SWITCHING 101 |
Switching protocols. | |
#define | URL_CODE_TEMPORARY_REDIRECT 307 |
The request should be repeated at another URI location. | |
#define | URL_CODE_UNAUTHORIZED 401 |
Authentication for the request has failed. | |
#define | URL_CODE_UNSUPPORTED_MEDIA_TYPE 415 |
The request media type is not supported by the server or resource. | |
#define | URL_CODE_USE_PROXY 305 |
The requested resource must be accessed via the location proxy. | |
#define | URL_NOTIFY_DONE 3 |
Request is complete. | |
#define | URL_NOTIFY_IO 1 |
Streaming I/O event. | |
#define | URL_NOTIFY_RETRY 2 |
Retry initiated. |
Url
Url request object.
- Description:
- The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.
- API Stability:
- Evolving.
- Fields:
-
uint certsDefined Certificates have been defined. uint chunked Request is using transfer chunk encoding. uint close Connection should be closed on completion of the current request. Ticks deadline Request time limit expiry deadline. char * error Error message. cchar * host Request host. cchar * path Request path without leading "/" and query/ref. int port Request port. cchar * query Request query portion. cchar * reference Request reference portion. RBuf * response Buffer to hold complete response. uint responseRead Response has been read. uint retries Client implemented retry field. RBuf * rx Buffer for reading responses. char * rxHeaders Rx headers. ssize rxLen Length of rx body. ssize rxRemaining Rx headers. cchar * scheme Request scheme. RSocket * sock Network socket. uint status Response (rx) status. Ticks timeout Request timeout. ssize txLen Length of tx body. char * urlbuf HTTP request method. Parsed and tokenized url. uint wroteHeaders Tx headers have been written.
Allocate a URL object.
- Description:
- A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.
- Returns:
- The url object.
- API Stability:
- Evolving.
- See Also:
Close the underlying network socket associated with the URL object.
- Parameters:
-
up URL object.
- API Stability:
- Evolving.
- See Also:
Fetch a URL.
- Description:
- This routine issues a HTTP request and reads and buffers the response. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. method HTTP method verb. uri HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. Set to 0 if none. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Response HTTP status code. Use urlGetResponse or urlRead to read the response.
- API Stability:
- Evolving.
- See Also:
Fetch a URL and return a JSON response.
- Description:
- This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. method HTTP method verb. uri HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. Set to 0 if none. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used. ... Optional header arguments.
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().
- API Stability:
- Evolving.
- See Also:
Free a URL object.
- Parameters:
-
up URL object.
- API Stability:
- Evolving.
- See Also:
Get a URL using a HTTP GET request.
- Description:
- This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
- Returns:
- Response body if successful, otherwise null. Caller must free.
- API Stability:
- Evolving.
- See Also:
Get the URL internal error message.
- Parameters:
-
up URL object.
- Returns:
- The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.
- API Stability:
- Evolving.
- See Also:
Get a response HTTP header.
- Parameters:
-
up URL object. header HTTP header name. This can be any case. For example: "Authorization".
- Returns:
- The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.
- API Stability:
- Evolving.
- See Also:
Get the response to a URL request as a JSON object tree.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object.
- Returns:
- The response body as parsed JSON object. Caller must free the result via jsonFree.
- API Stability:
- Evolving.
- See Also:
Get the response to a URL request.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object.
- Returns:
- The response body as a string. Caller must NOT free.
- API Stability:
- Evolving.
- See Also:
Get the HTTP response status code for a request.
- Parameters:
-
up URL object.
- Returns:
- The HTTP status code for the most recently completed request.
- API Stability:
- Evolving.
- See Also:
Parse a URL into its constituent components in the Url structure.
- Parameters:
-
up URL object. uri Uri to parse.
- Returns:
- Zero if the uri parses successfully.
- API Stability:
- Evolving.
- See Also:
Issue a HTTP POST request.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
- Returns:
- Response body if successful, otherwise null. Caller must free.
- API Stability:
- Evolving.
- See Also:
Issue a HTTP POST request and return parsed JSON.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. data Body data for request. Set to NULL if none. len Size of body data for request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
- API Stability:
- Evolving.
- See Also:
Read response data for a request.
- Description:
- This routine will block the current fiber if necessary. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. buf Buffer to read into. bufsize Size of the buffer.
- Returns:
- The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.
- API Stability:
- Evolving.
- See Also:
Define the certificates to use with TLS.
- Parameters:
-
up URL object. key Certificate private key. cert Certificate text. ca Certificate authority text. revoke Certificates to revoke.
- API Stability:
- Evolving.
- See Also:
Set the default number of retries for requests.
- Description:
- This does not change the number of retries for existing Url objects.
- Parameters:
-
retries Number of retries.
- API Stability:
- Evolving.
- See Also:
Set the default request timeout to use for future URL instances.
- Description:
- This does not change the timeout for existing Url objects.
- Parameters:
-
timeout Timeout in milliseconds.
- API Stability:
- Evolving.
- See Also:
Set the URL internal error message.
- Parameters:
-
up URL object. message Printf style message format string. ... Message arguments.
- API Stability:
- Evolving.
- See Also:
Set the request timeout to use for the specific URL object.
- Parameters:
-
up URL object. timeout Timeout in milliseconds.
- API Stability:
- Evolving.
- See Also:
Control verification of TLS connections.
- Parameters:
-
up URL object. verifyPeer Set to true to verify the certificate of the remote peer. verifyIssuer Set to true to verify the issuer of the peer certificate.
- API Stability:
- Evolving.
- See Also:
Start a Url request.
- Description:
- This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.
- Must only be called from a fiber.
- Parameters:
-
up URL object. method HTTP method verb. uri HTTP URL to fetch. txLen Content length of request body. This must match the length of data written with urlWrite. Set to -1 if the content length is not known.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
- See Also:
Write body data for a request.
- Description:
- This routine will block the current fiber. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. data Buffer of data to write. size Length of data to write.
- Returns:
- The number of bytes actually written.
- API Stability:
- Evolving.
- See Also:
Write request headers.
- Description:
- This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will block the current fiber. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
- See Also:
Typedefs
URL callback procedure.
- Parameters:
-
up URL object. event Type of event. Set to URL_NOTIFY_IO, URL_NOTIFY_RETRY or URL_NOTIFY_DONE. buf Associated data. len Associated data length.
- API Stability:
- Evolving.
- See Also: