Skip to content

URL API

The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.

The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.

The library has configurable request/response timeouts and retries.

Features:

  • Supports transfer-chunk encoding for dynamic request bodies.
  • Control over headers
  • Full-duplex streaming support for minimal memory usage.
  • Full support for TLS including certificate verification.
  • Configurable retries and timeouts.
  • Parallelism via fiber coroutines.

The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.

Extensions

Url Url request object.

Functions

Url*urlAlloc(void)
 Allocate a URL object.
voidurlClose(Url *up)
 Close the underlying network socket associated with the URL object.
inturlFetch(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)
 Fetch a URL.
Json*urlFetchJson(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)
 Fetch a URL and return a JSON response.
voidurlFree(Url *up)
 Free a URL object.
char*urlGet(cchar *uri, cchar *headers, ...)
 Get a URL using a HTTP GET request.
cchar*urlGetError(Url *up)
 Get the URL internal error message.
cchar*urlGetHeader(Url *up, cchar *header)
 Get a response HTTP header.
Json*urlGetJsonResponse(Url *up)
 Get the response to a URL request as a JSON object tree.
cchar*urlGetResponse(Url *up)
 Get the response to a URL request.
inturlGetStatus(Url *up)
 Get the HTTP response status code for a request.
inturlParse(Url *up, cchar *uri)
 Parse a URL into its constituent components in the Url structure.
char*urlPost(cchar *uri, cvoid *data, ssize size, cchar *headers, ...)
 Issue a HTTP POST request.
Json*urlPostJson(cchar *uri, cvoid *data, ssize len, cchar *headers, ...)
 Issue a HTTP POST request and return parsed JSON.
ssizeurlRead(Url *up, char *buf, ssize bufsize)
 Read response data for a request.
voidurlSetCerts(Url *up, cchar *key, cchar *cert, cchar *ca, cchar *revoke)
 Define the certificates to use with TLS.
voidurlSetDefaultRetries(int retries)
 Set the default number of retries for requests.
voidurlSetDefaultTimeout(int timeout)
 Set the default request timeout to use for future URL instances.
voidurlSetError(Url *up, cchar *message, ...)
 Set the URL internal error message.
voidurlSetTimeout(Url *up, int timeout)
 Set the request timeout to use for the specific URL object.
voidurlSetVerify(Url *up, int verifyPeer, int verifyIssuer)
 Control verification of TLS connections.
inturlStart(Url *up, cchar *method, cchar *uri, ssize txLen)
 Start a Url request.
ssizeurlWrite(Url *up, cvoid *data, ssize size)
 Write body data for a request.
inturlWriteHeaders(Url *up, cchar *headers, ...)
 Write request headers.

Typedefs

UrlProcURL callback procedure.

Defines

#defineURL_CODE_ACCEPTED   202
 The request has been accepted and processing is continuing.
#defineURL_CODE_BAD_GATEWAY   502
 The server cannot act as a gateway for the given request.
#defineURL_CODE_BAD_METHOD   405
 The request HTTP method was not supported by the resource.
#defineURL_CODE_BAD_REQUEST   400
 The request is malformed.
#defineURL_CODE_BAD_VERSION   505
 The server does not support the HTTP protocol version.
#defineURL_CODE_CONFLICT   409
 The request had a conflict in the request headers and URI.
#defineURL_CODE_CONTINUE   100
 Continue with request, only parial content transmitted.
#defineURL_CODE_CREATED   201
 The request has completed and a new resource was created.
#defineURL_CODE_EXPECTATION_FAILED   417
 The server cannot satisfy the Expect header requirements.
#defineURL_CODE_FORBIDDEN   403
 The request was legal, but the server refuses to process.
#defineURL_CODE_GATEWAY_TIMEOUT   504
 The server gateway timed out waiting for the upstream server.
#defineURL_CODE_GONE   410
 The requested resource is no longer available.
#defineURL_CODE_IM_A_TEAPOT   418
 Short and stout error code (RFC 2324).
#defineURL_CODE_INSUFFICIENT_STORAGE   507
 The server has insufficient storage to complete the request.
#defineURL_CODE_INTERNAL_SERVER_ERROR   500
 Server processing or configuration error.
#defineURL_CODE_LENGTH_REQUIRED   411
 The request did not specify a required content length.
#defineURL_CODE_MOVED_PERMANENTLY   301
 The requested URI has moved permanently to a new location.
#defineURL_CODE_MOVED_TEMPORARILY   302
 The URI has moved temporarily to a new location.
#defineURL_CODE_NO_CONTENT   204
 The request has completed and there is no response to send.
#defineURL_CODE_NO_RESPONSE   444
 The connection was closed with no response to the client.
#defineURL_CODE_NOT_ACCEPTABLE   406
 The requested resource cannot generate the required content.
#defineURL_CODE_NOT_AUTHORITATIVE   203
 The request has completed but content may be from another source.
#defineURL_CODE_NOT_FOUND   404
 The requested resource was not found.
#defineURL_CODE_NOT_IMPLEMENTED   501
 The server does not recognize the request or method.
#defineURL_CODE_NOT_MODIFIED   304
 The requested resource has changed since the last request.
#defineURL_CODE_OK   200
 The request completed successfully.
#defineURL_CODE_PARTIAL   206
 The request has completed and is returning parial content.
#defineURL_CODE_PAYMENT_REQUIRED   402
 Reserved for future use.
#defineURL_CODE_PRECOND_FAILED   412
 The server cannot satisfy one of the request preconditions.
#defineURL_CODE_RANGE_NOT_SATISFIABLE   416
 The request content range does not exist for the resource.
#defineURL_CODE_REQUEST_TIMEOUT   408
 The server timed out waiting for the request to complete.
#defineURL_CODE_REQUEST_TOO_LARGE   413
 The request is too large for the server to process.
#defineURL_CODE_REQUEST_URL_TOO_LARGE   414
 The request URI is too long for the server to process.
#defineURL_CODE_RESET   205
 The request has completed with no content.
#defineURL_CODE_SEE_OTHER   303
 The requested URI can be found at another URI location.
#defineURL_CODE_SERVICE_UNAVAILABLE   503
 The server is currently unavailable or overloaded.
#defineURL_CODE_SWITCHING   101
 Switching protocols.
#defineURL_CODE_TEMPORARY_REDIRECT   307
 The request should be repeated at another URI location.
#defineURL_CODE_UNAUTHORIZED   401
 Authentication for the request has failed.
#defineURL_CODE_UNSUPPORTED_MEDIA_TYPE   415
 The request media type is not supported by the server or resource.
#defineURL_CODE_USE_PROXY   305
 The requested resource must be accessed via the location proxy.
#defineURL_NOTIFY_DONE   3
 Request is complete.
#defineURL_NOTIFY_IO   1
 Streaming I/O event.
#defineURL_NOTIFY_RETRY   2
 Retry initiated.

Url

Url

Url request object.

Description:
The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.
API Stability:
Evolving.
Fields:
uintcertsDefined Certificates have been defined.
uintchunked Request is using transfer chunk encoding.
uintclose Connection should be closed on completion of the current request.
Ticksdeadline Request time limit expiry deadline.
char *error Error message.
cchar *host Request host.
cchar *path Request path without leading "/" and query/ref.
intport Request port.
cchar *query Request query portion.
cchar *reference Request reference portion.
RBuf *response Buffer to hold complete response.
uintresponseRead Response has been read.
uintretries Client implemented retry field.
RBuf *rx Buffer for reading responses.
char *rxHeaders Rx headers.
ssizerxLen Length of rx body.
ssizerxRemaining Rx headers.
cchar *scheme Request scheme.
RSocket *sock Network socket.
uintstatus Response (rx) status.
Tickstimeout Request timeout.
ssizetxLen Length of tx body.
char *urlbuf HTTP request method. Parsed and tokenized url.
uintwroteHeaders Tx headers have been written.

Url * urlAlloc (void )

Allocate a URL object.

Description:
A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.
Returns:
The url object.
API Stability:
Evolving.
See Also:

void urlClose (Url *up)

Close the underlying network socket associated with the URL object.

Parameters:
upURL object.
API Stability:
Evolving.
See Also:

int urlFetch (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL.

Description:
This routine issues a HTTP request and reads and buffers the response. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
methodHTTP method verb.
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
sizeSize of body data for request. Set to 0 if none.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Optional header arguments.
Returns:
Response HTTP status code. Use urlGetResponse or urlRead to read the response.
API Stability:
Evolving.
See Also:

Json * urlFetchJson (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL and return a JSON response.

Description:
This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
methodHTTP method verb.
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
sizeSize of body data for request. Set to 0 if none.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used.
...Optional header arguments.
Returns:
Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().
API Stability:
Evolving.
See Also:

void urlFree (Url *up)

Free a URL object.

Parameters:
upURL object.
API Stability:
Evolving.
See Also:

char * urlGet (cchar *uri, cchar *headers, ...)

Get a URL using a HTTP GET request.

Description:
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
Returns:
Response body if successful, otherwise null. Caller must free.
API Stability:
Evolving.
See Also:

cchar * urlGetError (Url *up)

Get the URL internal error message.

Parameters:
upURL object.
Returns:
The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.
API Stability:
Evolving.
See Also:

cchar * urlGetHeader (Url *up, cchar *header)

Get a response HTTP header.

Parameters:
upURL object.
headerHTTP header name. This can be any case. For example: "Authorization".
Returns:
The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.
API Stability:
Evolving.
See Also:

Json * urlGetJsonResponse (Url *up)

Get the response to a URL request as a JSON object tree.

Description:
After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
Returns:
The response body as parsed JSON object. Caller must free the result via jsonFree.
API Stability:
Evolving.
See Also:

cchar * urlGetResponse (Url *up)

Get the response to a URL request.

Description:
After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
Returns:
The response body as a string. Caller must NOT free.
API Stability:
Evolving.
See Also:

int urlGetStatus (Url *up)

Get the HTTP response status code for a request.

Parameters:
upURL object.
Returns:
The HTTP status code for the most recently completed request.
API Stability:
Evolving.
See Also:

int urlParse (Url *up, cchar *uri)

Parse a URL into its constituent components in the Url structure.

Parameters:
upURL object.
uriUri to parse.
Returns:
Zero if the uri parses successfully.
API Stability:
Evolving.
See Also:

char * urlPost (cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Issue a HTTP POST request.

Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
sizeSize of body data for request.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
Returns:
Response body if successful, otherwise null. Caller must free.
API Stability:
Evolving.
See Also:

Json * urlPostJson (cchar *uri, cvoid *data, ssize len, cchar *headers, ...)

Issue a HTTP POST request and return parsed JSON.

Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
lenSize of body data for request.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
Returns:
Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
API Stability:
Evolving.
See Also:

ssize urlRead (Url *up, char *buf, ssize bufsize)

Read response data for a request.

Description:
This routine will block the current fiber if necessary. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
bufBuffer to read into.
bufsizeSize of the buffer.
Returns:
The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.
API Stability:
Evolving.
See Also:

void urlSetCerts (Url *up, cchar *key, cchar *cert, cchar *ca, cchar *revoke)

Define the certificates to use with TLS.

Parameters:
upURL object.
keyCertificate private key.
certCertificate text.
caCertificate authority text.
revokeCertificates to revoke.
API Stability:
Evolving.
See Also:

void urlSetDefaultRetries (int retries)

Set the default number of retries for requests.

Description:
This does not change the number of retries for existing Url objects.
Parameters:
retriesNumber of retries.
API Stability:
Evolving.
See Also:

void urlSetDefaultTimeout (int timeout)

Set the default request timeout to use for future URL instances.

Description:
This does not change the timeout for existing Url objects.
Parameters:
timeoutTimeout in milliseconds.
API Stability:
Evolving.
See Also:

void urlSetError (Url *up, cchar *message, ...)

Set the URL internal error message.

Parameters:
upURL object.
messagePrintf style message format string.
...Message arguments.
API Stability:
Evolving.
See Also:

void urlSetTimeout (Url *up, int timeout)

Set the request timeout to use for the specific URL object.

Parameters:
upURL object.
timeoutTimeout in milliseconds.
API Stability:
Evolving.
See Also:

void urlSetVerify (Url *up, int verifyPeer, int verifyIssuer)

Control verification of TLS connections.

Parameters:
upURL object.
verifyPeerSet to true to verify the certificate of the remote peer.
verifyIssuerSet to true to verify the issuer of the peer certificate.
API Stability:
Evolving.
See Also:

int urlStart (Url *up, cchar *method, cchar *uri, ssize txLen)

Start a Url request.

Description:
This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.
Must only be called from a fiber.
Parameters:
upURL object.
methodHTTP method verb.
uriHTTP URL to fetch.
txLenContent length of request body. This must match the length of data written with urlWrite. Set to -1 if the content length is not known.
Returns:
Zero if successful.
API Stability:
Evolving.
See Also:

ssize urlWrite (Url *up, cvoid *data, ssize size)

Write body data for a request.

Description:
This routine will block the current fiber. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
dataBuffer of data to write.
sizeLength of data to write.
Returns:
The number of bytes actually written.
API Stability:
Evolving.
See Also:

int urlWriteHeaders (Url *up, cchar *headers, ...)

Write request headers.

Description:
This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will block the current fiber. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Optional header arguments.
Returns:
Zero if successful.
API Stability:
Evolving.
See Also:

Typedefs

typedef void(* UrlProc) (struct Url *up, int event, char *buf, ssize len).

URL callback procedure.

Parameters:
upURL object.
eventType of event. Set to URL_NOTIFY_IO, URL_NOTIFY_RETRY or URL_NOTIFY_DONE.
bufAssociated data.
lenAssociated data length.
API Stability:
Evolving.
See Also: