URL API
The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.
The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.
The library has configurable request/response timeouts and retries.
Features:
- Supports transfer-chunk encoding for dynamic request bodies.
- Control over headers
- Full-duplex streaming support for minimal memory usage.
- Full support for TLS including certificate verification.
- Configurable retries and timeouts.
- Parallelism via fiber coroutines.
The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.
Extensions
Url | Url request object. |
Functions
Url * | urlAlloc(int flags) |
Allocate a URL object. | |
void | urlClose(Url *up) |
Close the underlying network socket associated with the URL object. | |
int | urlFetch(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...) |
Fetch a URL. | |
ssize | urlFinalize(Url *url) |
Finalize request body. | |
void | urlFree(Url *up) |
Free a URL object. | |
char * | urlGet(cchar *uri, cchar *headers, ...) |
Get a URL using a HTTP GET request. | |
cchar * | urlGetError(Url *up) |
Get the URL internal error message. | |
cchar * | urlGetHeader(Url *up, cchar *header) |
Get a response HTTP header from the parsed response headers. | |
Json * | urlGetJson(cchar *uri, cchar *headers, ...) |
Issue a HTTP GET request and return parsed JSON. | |
Json * | urlGetJsonResponse(Url *up) |
Get the response to a URL request as a JSON object tree. | |
cchar * | urlGetResponse(Url *up) |
Get the response to a URL request as a string. | |
RBuf * | urlGetResponseBuf(Url *up) |
Get the response to a URL request in a buffer. | |
int | urlGetStatus(Url *up) |
Get the HTTP response status code for a request. | |
Json * | urlJson(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...) |
Fetch a URL and return a JSON response. | |
int | urlParse(Url *up, cchar *uri) |
Parse a URL into its constituent components in the Url structure. | |
char * | urlPost(cchar *uri, cvoid *data, ssize size, cchar *headers, ...) |
Issue a HTTP POST request. | |
Json * | urlPostJson(cchar *uri, cvoid *data, ssize len, cchar *headers, ...) |
Issue a HTTP POST request and return parsed JSON. | |
ssize | urlRead(Url *up, char *buf, ssize bufsize) |
Low level read routine to read response data for a request. | |
void | urlSetCerts(Url *up, cchar *ca, cchar *key, cchar *cert, cchar *revoke) |
Define the certificates to use with TLS. | |
void | urlSetCiphers(Url *up, cchar *ciphers) |
Set the list of available ciphers to use. | |
void | urlSetDefaultRetries(int retries) |
Set the default number of retries for requests. | |
void | urlSetDefaultTimeout(Ticks timeout) |
Set the default request timeout to use for future URL instances. | |
void | urlSetError(Url *up, cchar *message, ...) |
Set the URL internal error message. | |
void | urlSetFlags(Url *up, int flags) |
Set the URL flags. | |
void | urlSetProtocol(Url *up, int protocol) |
Set the HTTP protocol to use. | |
void | urlSetStatusError(Url *up, int status, cchar *message, ...) |
Set the URL internal error message and HTTP response status. | |
void | urlSetTimeout(Url *up, Ticks timeout) |
Set the request timeout to use for the specific URL object. | |
void | urlSetVerify(Url *up, int verifyPeer, int verifyIssuer) |
Control verification of TLS connections. | |
int | urlStart(Url *up, cchar *method, cchar *uri, ssize txLen) |
Start a Url request. | |
int | urlUpload(Url *up, RList *files, RHash *forms, cchar *headers, ...) |
Upload files in a request. | |
ssize | urlWrite(Url *up, cvoid *data, ssize size) |
Write body data for a request. | |
ssize | urlWriteFile(Url *up, cchar *path) |
Write a file contents for a request. | |
ssize | urlWriteFmt(Url *up, cchar *fmt, ...) |
Write formatted body data for a request. | |
int | urlWriteHeaders(Url *up, cchar *headers, ...) |
Write request headers. |
Typedefs
UrlProc | URL callback procedure. |
Defines
#define | URL_CODE_ACCEPTED 202 |
The request has been accepted and processing is continuing. | |
#define | URL_CODE_BAD_GATEWAY 502 |
The server cannot act as a gateway for the given request. | |
#define | URL_CODE_BAD_METHOD 405 |
The request HTTP method was not supported by the resource. | |
#define | URL_CODE_BAD_REQUEST 400 |
The request is malformed. | |
#define | URL_CODE_BAD_VERSION 505 |
The server does not support the HTTP protocol version. | |
#define | URL_CODE_CONFLICT 409 |
The request had a conflict in the request headers and URI. | |
#define | URL_CODE_CONTINUE 100 |
Continue with request, only parial content transmitted. | |
#define | URL_CODE_CREATED 201 |
The request has completed and a new resource was created. | |
#define | URL_CODE_EXPECTATION_FAILED 417 |
The server cannot satisfy the Expect header requirements. | |
#define | URL_CODE_FORBIDDEN 403 |
The request was legal, but the server refuses to process. | |
#define | URL_CODE_GATEWAY_TIMEOUT 504 |
The server gateway timed out waiting for the upstream server. | |
#define | URL_CODE_GONE 410 |
The requested resource is no longer available. | |
#define | URL_CODE_IM_A_TEAPOT 418 |
Short and stout error code (RFC 2324). | |
#define | URL_CODE_INSUFFICIENT_STORAGE 507 |
The server has insufficient storage to complete the request. | |
#define | URL_CODE_INTERNAL_SERVER_ERROR 500 |
Server processing or configuration error. | |
#define | URL_CODE_LENGTH_REQUIRED 411 |
The request did not specify a required content length. | |
#define | URL_CODE_MOVED_PERMANENTLY 301 |
The requested URI has moved permanently to a new location. | |
#define | URL_CODE_MOVED_TEMPORARILY 302 |
The URI has moved temporarily to a new location. | |
#define | URL_CODE_NO_CONTENT 204 |
The request has completed and there is no response to send. | |
#define | URL_CODE_NO_RESPONSE 444 |
The connection was closed with no response to the client. | |
#define | URL_CODE_NOT_ACCEPTABLE 406 |
The requested resource cannot generate the required content. | |
#define | URL_CODE_NOT_AUTHORITATIVE 203 |
The request has completed but content may be from another source. | |
#define | URL_CODE_NOT_FOUND 404 |
The requested resource was not found. | |
#define | URL_CODE_NOT_IMPLEMENTED 501 |
The server does not recognize the request or method. | |
#define | URL_CODE_NOT_MODIFIED 304 |
The requested resource has changed since the last request. | |
#define | URL_CODE_OK 200 |
The request completed successfully. | |
#define | URL_CODE_PARTIAL 206 |
The request has completed and is returning parial content. | |
#define | URL_CODE_PAYMENT_REQUIRED 402 |
Reserved for future use. | |
#define | URL_CODE_PRECOND_FAILED 412 |
The server cannot satisfy one of the request preconditions. | |
#define | URL_CODE_RANGE_NOT_SATISFIABLE 416 |
The request content range does not exist for the resource. | |
#define | URL_CODE_REQUEST_TIMEOUT 408 |
The server timed out waiting for the request to complete. | |
#define | URL_CODE_REQUEST_TOO_LARGE 413 |
The request is too large for the server to process. | |
#define | URL_CODE_REQUEST_URL_TOO_LARGE 414 |
The request URI is too long for the server to process. | |
#define | URL_CODE_RESET 205 |
The request has completed with no content. | |
#define | URL_CODE_SEE_OTHER 303 |
The requested URI can be found at another URI location. | |
#define | URL_CODE_SERVICE_UNAVAILABLE 503 |
The server is currently unavailable or overloaded. | |
#define | URL_CODE_SWITCHING 101 |
Switching protocols. | |
#define | URL_CODE_TEMPORARY_REDIRECT 307 |
The request should be repeated at another URI location. | |
#define | URL_CODE_UNAUTHORIZED 401 |
Authentication for the request has failed. | |
#define | URL_CODE_UNSUPPORTED_MEDIA_TYPE 415 |
The request media type is not supported by the server or resource. | |
#define | URL_CODE_USE_PROXY 305 |
The requested resource must be accessed via the location proxy. | |
#define | URL_HTTP_0 0x20 |
Use HTTP/1.0. | |
#define | URL_NOTIFY_DONE 3 |
Request is complete. | |
#define | URL_NOTIFY_IO 1 |
Streaming I/O event. | |
#define | URL_NOTIFY_RETRY 2 |
Retry initiated. | |
#define | URL_SHOW_NONE 0x1 |
Trace nothing. | |
#define | URL_SHOW_REQ_BODY 0x2 |
Trace request body. | |
#define | URL_SHOW_REQ_HEADERS 0x4 |
Trace request headers. | |
#define | URL_SHOW_RESP_BODY 0x8 |
Trace response body. | |
#define | URL_SHOW_RESP_HEADERS 0x10 |
Trace response headers. |
Url
Url request object.
- Description:
- The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.
- API Stability:
- Evolving.
- Fields:
-
char * boundary Multipart mime upload file boundary. uint certsDefined Certificates have been defined. uint chunked Request is using transfer chunk encoding. uint close Connection should be closed on completion of the current request. Ticks deadline Request time limit expiry deadline. char * error Error message. uint flags Alloc flags. uint gotResponse Response has been read. cchar * hash Request hash portion. cchar * host Request host. cchar * path Request path without leading "/" and query/ref. int port Request port. uint protocol Use HTTP/1.0 without keep-alive. Defaults to HTTP/1.1. cchar * query Request query portion. cchar * redirect Redirect location. char * response Response as a string. RBuf * responseBuf Buffer to hold complete response. uint retries Client implemented retry field. RBuf * rx Buffer for progressive reading response data. RBuf * rxHeaders Buffer for Rx headers. ssize rxLen Length of rx body. ssize rxRemaining Remaining rx data to read from the socket. cchar * scheme Request scheme. RSocket * sock Network socket. uint status Response (rx) status. Ticks timeout Request timeout. ssize txLen Length of tx body. char * urlbuf HTTP request method. Parsed and tokenized url. uint wroteHeaders Tx headers have been written.
Allocate a URL object.
- Description:
- A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.
- Parameters:
-
flags Set flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY. Defaults to 0.
- Returns:
- The url object.
- API Stability:
- Evolving.
Close the underlying network socket associated with the URL object.
- Description:
- This is not normally necessary to invoke unless you want to force the socket connection to be recreated before issuing a subsequent request.
- Parameters:
-
up URL object.
- API Stability:
- Evolving.
Fetch a URL.
- Description:
- This routine issues a HTTP request with optional body data and returns the HTTP status code. This routine will return before the response body data has been read. Use $urlRead or $urlGetResponse to read the response body data. This routine will block the current fiber while waiting for the request to respond. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. method HTTP method verb. uri HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. Set to 0 if none. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Response HTTP status code. Use urlGetResponse or urlRead to read the response.
- API Stability:
- Evolving.
Finalize request body.
- Description:
- This routine will call urlWrite(web, NULL, 0);.
- Parameters:
-
url Url object.
- Returns:
- Number of bytes written. Returns a negative status code on errors.
- API Stability:
- Evolving.
Free a URL object.
- Parameters:
-
up URL object.
- API Stability:
- Evolving.
Get a URL using a HTTP GET request.
- Description:
- This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Response body if successful, otherwise null. Caller must free.
- API Stability:
- Evolving.
Get the URL internal error message.
- Description:
- Errors are defined for unexpected errors. HTTP requests that respond with a non-200 status do not set the error message.
- Parameters:
-
up URL object.
- Returns:
- The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.
- API Stability:
- Evolving.
Get a response HTTP header from the parsed response headers.
- Parameters:
-
up URL object. header HTTP header name. This can be any case. For example: "Authorization".
- Returns:
- The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.
- API Stability:
- Evolving.
Issue a HTTP GET request and return parsed JSON.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
- API Stability:
- Evolving.
Get the response to a URL request as a JSON object tree.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object.
- Returns:
- The response body as parsed JSON object. Caller must free the result via jsonFree.
- API Stability:
- Evolving.
Get the response to a URL request as a string.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
If receiving a binary response, use urlGetResponseBuf instead.
- Must only be called from a fiber.
- Parameters:
-
up URL object.
- Returns:
- The response body as a string. Caller must NOT free.
- API Stability:
- Evolving.
Get the response to a URL request in a buffer.
- Description:
- After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object.
- Returns:
- The response body as runtime buffer. Caller must NOT free.
- API Stability:
- Evolving.
Get the HTTP response status code for a request.
- Parameters:
-
up URL object.
- Returns:
- The HTTP status code for the most recently completed request.
- API Stability:
- Evolving.
Fetch a URL and return a JSON response.
- Description:
- This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. method HTTP method verb. uri HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. Set to 0 if none. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used. ... Optional header arguments.
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().
- API Stability:
- Evolving.
Parse a URL into its constituent components in the Url structure.
- Parameters:
-
up URL object. uri Uri to parse.
- Returns:
- Zero if the uri parses successfully.
- API Stability:
- Evolving.
Issue a HTTP POST request.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. data Body data for request. Set to NULL if none. size Size of body data for request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Headers arguments.
- Returns:
- Response body if successful, otherwise null. Caller must free.
- API Stability:
- Evolving.
Issue a HTTP POST request and return parsed JSON.
- Must only be called from a fiber.
- Parameters:
-
uri HTTP URL to fetch. data Body data for request. Set to NULL if none. len Size of body data for request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
- Returns:
- Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
- API Stability:
- Evolving.
Low level read routine to read response data for a request.
- Description:
- This routine may be called to progressively read response data. It should not be called if using urlGetResponse directly or indirectly via urlGet, urlPost, urlPostJson or urlJson. This routine will block the current fiber if necessary. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. buf Buffer to read into. bufsize Size of the buffer.
- Returns:
- The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.
- API Stability:
- Evolving.
Define the certificates to use with TLS.
- Parameters:
-
up URL object. ca Certificate authority to verify client certificates. key Certificate private key. cert Certificate text. revoke Certificates to revoke.
- API Stability:
- Evolving.
Set the list of available ciphers to use.
- Parameters:
-
up URL object. ciphers String list of available ciphers.
- API Stability:
- Prototype.
Set the default number of retries for requests.
- Description:
- This does not change the number of retries for existing Url objects.
- Parameters:
-
retries Number of retries.
- API Stability:
- Evolving.
Set the default request timeout to use for future URL instances.
- Description:
- This does not change the timeout for existing Url objects.
- Parameters:
-
timeout Timeout in milliseconds.
- API Stability:
- Evolving.
Set the URL internal error message.
- Parameters:
-
up URL object. message Printf style message format string. ... Message arguments.
- API Stability:
- Evolving.
Set the URL flags.
- Parameters:
-
up URL object. flags Set flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY.
- API Stability:
- Prototype.
Set the HTTP protocol to use.
- Parameters:
-
up URL object. protocol Set to 0 for HTTP/1.0 or 1 for HTTP/1.1. Defaults to 1.
- API Stability:
- Prototype.
Set the URL internal error message and HTTP response status.
- Parameters:
-
up URL object. status HTTP status code. message Printf style message format string. ... Message arguments.
- API Stability:
- Evolving.
Set the request timeout to use for the specific URL object.
- Parameters:
-
up URL object. timeout Timeout in milliseconds.
- API Stability:
- Evolving.
Control verification of TLS connections.
- Parameters:
-
up URL object. verifyPeer Set to true to verify the certificate of the remote peer. verifyIssuer Set to true to verify the issuer of the peer certificate.
- API Stability:
- Evolving.
Start a Url request.
- Description:
- This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.
- Must only be called from a fiber.
- Parameters:
-
up URL object. method HTTP method verb. uri HTTP URL to fetch. txLen Content length of request body. This must match the length of data written with urlWrite. Set to -1 if the content length is not known.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Upload files in a request.
- Description:
- This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.
- Must only be called from a fiber.
- Parameters:
-
up URL object. files List of filenames to upload. forms Hash of key/value form values to add to the request. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Headers arguments.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Write body data for a request.
- Description:
- This routine will block the current fiber. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. data Buffer of data to write. size Length of data to write. Set to -1 to calculate the length of data as a null terminated string.
- Returns:
- The number of bytes actually written. On errors, returns a negative status code.
- API Stability:
- Evolving.
Write a file contents for a request.
- Description:
- This routine will read the file contents and write to the client. It will block the current fiber. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. path File pathname.
- Returns:
- The number of bytes actually written. On errors, returns a negative status code.
- API Stability:
- Evolving.
Write formatted body data for a request.
- Description:
- This routine will block the current fiber. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. fmt Printf style format string. ... Format arguments.
- Returns:
- The number of bytes actually written. On errors, returns a negative status code.
- API Stability:
- Evolving.
Write request headers.
- Description:
- This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will block the current fiber. Other fibers continue to run.
- Must only be called from a fiber.
- Parameters:
-
up URL object. headers Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". ... Optional header arguments.
- Returns:
- Zero if successful.
- API Stability:
- Evolving.
Typedefs
URL callback procedure.
- Parameters:
-
up URL object. event Type of event. Set to URL_NOTIFY_IO, URL_NOTIFY_RETRY or URL_NOTIFY_DONE. buf Associated data. len Associated data length.
- API Stability:
- Evolving.