Skip to content

URL API

The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.

The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.

The library has configurable request/response timeouts and retries.

Features:

  • Supports transfer-chunk encoding for dynamic request bodies.
  • Control over headers
  • Full-duplex streaming support for minimal memory usage.
  • Full support for TLS including certificate verification.
  • Configurable retries and timeouts.
  • Parallelism via fiber coroutines.

The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.

Extensions

Url Url request object.

Functions

Url *urlAlloc(int flags)
 Allocate a URL object.
voidurlClose(Url *up)
 Close the underlying network socket associated with the URL object.
inturlFetch(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)
 Fetch a URL.
ssizeurlFinalize(Url *url)
 Finalize request body.
voidurlFree(Url *up)
 Free a URL object.
char *urlGet(cchar *uri, cchar *headers, ...)
 Get a URL using a HTTP GET request.
cchar *urlGetError(Url *up)
 Get the URL internal error message.
cchar *urlGetHeader(Url *up, cchar *header)
 Get a response HTTP header from the parsed response headers.
Json *urlGetJson(cchar *uri, cchar *headers, ...)
 Issue a HTTP GET request and return parsed JSON.
Json *urlGetJsonResponse(Url *up)
 Get the response to a URL request as a JSON object tree.
cchar *urlGetResponse(Url *up)
 Get the response to a URL request as a string.
RBuf *urlGetResponseBuf(Url *up)
 Get the response to a URL request in a buffer.
inturlGetStatus(Url *up)
 Get the HTTP response status code for a request.
Json *urlJson(Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)
 Fetch a URL and return a JSON response.
inturlParse(Url *up, cchar *uri)
 Parse a URL into its constituent components in the Url structure.
char *urlPost(cchar *uri, cvoid *data, ssize size, cchar *headers, ...)
 Issue a HTTP POST request.
Json *urlPostJson(cchar *uri, cvoid *data, ssize len, cchar *headers, ...)
 Issue a HTTP POST request and return parsed JSON.
ssizeurlRead(Url *up, char *buf, ssize bufsize)
 Low level read routine to read response data for a request.
voidurlSetCerts(Url *up, cchar *ca, cchar *key, cchar *cert, cchar *revoke)
 Define the certificates to use with TLS.
voidurlSetCiphers(Url *up, cchar *ciphers)
 Set the list of available ciphers to use.
voidurlSetDefaultRetries(int retries)
 Set the default number of retries for requests.
voidurlSetDefaultTimeout(Ticks timeout)
 Set the default request timeout to use for future URL instances.
voidurlSetError(Url *up, cchar *message, ...)
 Set the URL internal error message.
voidurlSetFlags(Url *up, int flags)
 Set the URL flags.
voidurlSetProtocol(Url *up, int protocol)
 Set the HTTP protocol to use.
voidurlSetStatusError(Url *up, int status, cchar *message, ...)
 Set the URL internal error message and HTTP response status.
voidurlSetTimeout(Url *up, Ticks timeout)
 Set the request timeout to use for the specific URL object.
voidurlSetVerify(Url *up, int verifyPeer, int verifyIssuer)
 Control verification of TLS connections.
inturlStart(Url *up, cchar *method, cchar *uri, ssize txLen)
 Start a Url request.
inturlUpload(Url *up, RList *files, RHash *forms, cchar *headers, ...)
 Upload files in a request.
ssizeurlWrite(Url *up, cvoid *data, ssize size)
 Write body data for a request.
ssizeurlWriteFile(Url *up, cchar *path)
 Write a file contents for a request.
ssizeurlWriteFmt(Url *up, cchar *fmt, ...)
 Write formatted body data for a request.
inturlWriteHeaders(Url *up, cchar *headers, ...)
 Write request headers.

Typedefs

UrlProcURL callback procedure.

Defines

#defineURL_CODE_ACCEPTED   202
 The request has been accepted and processing is continuing.
#defineURL_CODE_BAD_GATEWAY   502
 The server cannot act as a gateway for the given request.
#defineURL_CODE_BAD_METHOD   405
 The request HTTP method was not supported by the resource.
#defineURL_CODE_BAD_REQUEST   400
 The request is malformed.
#defineURL_CODE_BAD_VERSION   505
 The server does not support the HTTP protocol version.
#defineURL_CODE_CONFLICT   409
 The request had a conflict in the request headers and URI.
#defineURL_CODE_CONTINUE   100
 Continue with request, only parial content transmitted.
#defineURL_CODE_CREATED   201
 The request has completed and a new resource was created.
#defineURL_CODE_EXPECTATION_FAILED   417
 The server cannot satisfy the Expect header requirements.
#defineURL_CODE_FORBIDDEN   403
 The request was legal, but the server refuses to process.
#defineURL_CODE_GATEWAY_TIMEOUT   504
 The server gateway timed out waiting for the upstream server.
#defineURL_CODE_GONE   410
 The requested resource is no longer available.
#defineURL_CODE_IM_A_TEAPOT   418
 Short and stout error code (RFC 2324).
#defineURL_CODE_INSUFFICIENT_STORAGE   507
 The server has insufficient storage to complete the request.
#defineURL_CODE_INTERNAL_SERVER_ERROR   500
 Server processing or configuration error.
#defineURL_CODE_LENGTH_REQUIRED   411
 The request did not specify a required content length.
#defineURL_CODE_MOVED_PERMANENTLY   301
 The requested URI has moved permanently to a new location.
#defineURL_CODE_MOVED_TEMPORARILY   302
 The URI has moved temporarily to a new location.
#defineURL_CODE_NO_CONTENT   204
 The request has completed and there is no response to send.
#defineURL_CODE_NO_RESPONSE   444
 The connection was closed with no response to the client.
#defineURL_CODE_NOT_ACCEPTABLE   406
 The requested resource cannot generate the required content.
#defineURL_CODE_NOT_AUTHORITATIVE   203
 The request has completed but content may be from another source.
#defineURL_CODE_NOT_FOUND   404
 The requested resource was not found.
#defineURL_CODE_NOT_IMPLEMENTED   501
 The server does not recognize the request or method.
#defineURL_CODE_NOT_MODIFIED   304
 The requested resource has changed since the last request.
#defineURL_CODE_OK   200
 The request completed successfully.
#defineURL_CODE_PARTIAL   206
 The request has completed and is returning parial content.
#defineURL_CODE_PAYMENT_REQUIRED   402
 Reserved for future use.
#defineURL_CODE_PRECOND_FAILED   412
 The server cannot satisfy one of the request preconditions.
#defineURL_CODE_RANGE_NOT_SATISFIABLE   416
 The request content range does not exist for the resource.
#defineURL_CODE_REQUEST_TIMEOUT   408
 The server timed out waiting for the request to complete.
#defineURL_CODE_REQUEST_TOO_LARGE   413
 The request is too large for the server to process.
#defineURL_CODE_REQUEST_URL_TOO_LARGE   414
 The request URI is too long for the server to process.
#defineURL_CODE_RESET   205
 The request has completed with no content.
#defineURL_CODE_SEE_OTHER   303
 The requested URI can be found at another URI location.
#defineURL_CODE_SERVICE_UNAVAILABLE   503
 The server is currently unavailable or overloaded.
#defineURL_CODE_SWITCHING   101
 Switching protocols.
#defineURL_CODE_TEMPORARY_REDIRECT   307
 The request should be repeated at another URI location.
#defineURL_CODE_UNAUTHORIZED   401
 Authentication for the request has failed.
#defineURL_CODE_UNSUPPORTED_MEDIA_TYPE   415
 The request media type is not supported by the server or resource.
#defineURL_CODE_USE_PROXY   305
 The requested resource must be accessed via the location proxy.
#defineURL_HTTP_0   0x20
 Use HTTP/1.0.
#defineURL_NOTIFY_DONE   3
 Request is complete.
#defineURL_NOTIFY_IO   1
 Streaming I/O event.
#defineURL_NOTIFY_RETRY   2
 Retry initiated.
#defineURL_SHOW_NONE   0x1
 Trace nothing.
#defineURL_SHOW_REQ_BODY   0x2
 Trace request body.
#defineURL_SHOW_REQ_HEADERS   0x4
 Trace request headers.
#defineURL_SHOW_RESP_BODY   0x8
 Trace response body.
#defineURL_SHOW_RESP_HEADERS   0x10
 Trace response headers.

Url

Url

Url request object.

Description:
The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.
API Stability:
Evolving.
Fields:
char *boundary Multipart mime upload file boundary.
uintcertsDefined Certificates have been defined.
uintchunked Request is using transfer chunk encoding.
uintclose Connection should be closed on completion of the current request.
Ticksdeadline Request time limit expiry deadline.
char *error Error message.
uintflags Alloc flags.
uintgotResponse Response has been read.
cchar *hash Request hash portion.
cchar *host Request host.
cchar *path Request path without leading "/" and query/ref.
intport Request port.
uintprotocol Use HTTP/1.0 without keep-alive. Defaults to HTTP/1.1.
cchar *query Request query portion.
cchar *redirect Redirect location.
char *response Response as a string.
RBuf *responseBuf Buffer to hold complete response.
uintretries Client implemented retry field.
RBuf *rx Buffer for progressive reading response data.
RBuf *rxHeaders Buffer for Rx headers.
ssizerxLen Length of rx body.
ssizerxRemaining Remaining rx data to read from the socket.
cchar *scheme Request scheme.
RSocket *sock Network socket.
uintstatus Response (rx) status.
Tickstimeout Request timeout.
ssizetxLen Length of tx body.
char *urlbuf HTTP request method. Parsed and tokenized url.
uintwroteHeaders Tx headers have been written.

Url * * urlAlloc (int flags)

Allocate a URL object.

Description:
A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.
Parameters:
flagsSet flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY. Defaults to 0.
Returns:
The url object.
API Stability:
Evolving.

void urlClose (Url *up)

Close the underlying network socket associated with the URL object.

Description:
This is not normally necessary to invoke unless you want to force the socket connection to be recreated before issuing a subsequent request.
Parameters:
upURL object.
API Stability:
Evolving.

int urlFetch (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL.

Description:
This routine issues a HTTP request with optional body data and returns the HTTP status code. This routine will return before the response body data has been read. Use $urlRead or $urlGetResponse to read the response body data. This routine will block the current fiber while waiting for the request to respond. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
methodHTTP method verb.
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
sizeSize of body data for request. Set to 0 if none.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Optional header arguments.
Returns:
Response HTTP status code. Use urlGetResponse or urlRead to read the response.
API Stability:
Evolving.

ssize urlFinalize (Url *url)

Finalize request body.

Description:
This routine will call urlWrite(web, NULL, 0);.
Parameters:
urlUrl object.
Returns:
Number of bytes written. Returns a negative status code on errors.
API Stability:
Evolving.

void urlFree (Url *up)

Free a URL object.

Parameters:
upURL object.
API Stability:
Evolving.

char * * urlGet (cchar *uri, cchar *headers, ...)

Get a URL using a HTTP GET request.

Description:
This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Optional header arguments.
Returns:
Response body if successful, otherwise null. Caller must free.
API Stability:
Evolving.

cchar * * urlGetError (Url *up)

Get the URL internal error message.

Description:
Errors are defined for unexpected errors. HTTP requests that respond with a non-200 status do not set the error message.
Parameters:
upURL object.
Returns:
The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.
API Stability:
Evolving.

cchar * * urlGetHeader (Url *up, cchar *header)

Get a response HTTP header from the parsed response headers.

Parameters:
upURL object.
headerHTTP header name. This can be any case. For example: "Authorization".
Returns:
The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.
API Stability:
Evolving.

Json * * urlGetJson (cchar *uri, cchar *headers, ...)

Issue a HTTP GET request and return parsed JSON.

Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Optional header arguments.
Returns:
Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
API Stability:
Evolving.

Json * * urlGetJsonResponse (Url *up)

Get the response to a URL request as a JSON object tree.

Description:
After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
Returns:
The response body as parsed JSON object. Caller must free the result via jsonFree.
API Stability:
Evolving.

cchar * * urlGetResponse (Url *up)

Get the response to a URL request as a string.

Description:
After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

If receiving a binary response, use urlGetResponseBuf instead.
Must only be called from a fiber.
Parameters:
upURL object.
Returns:
The response body as a string. Caller must NOT free.
API Stability:
Evolving.

RBuf * * urlGetResponseBuf (Url *up)

Get the response to a URL request in a buffer.

Description:
After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
Returns:
The response body as runtime buffer. Caller must NOT free.
API Stability:
Evolving.

int urlGetStatus (Url *up)

Get the HTTP response status code for a request.

Parameters:
upURL object.
Returns:
The HTTP status code for the most recently completed request.
API Stability:
Evolving.

Json * * urlJson (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL and return a JSON response.

Description:
This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
methodHTTP method verb.
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
sizeSize of body data for request. Set to 0 if none.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used.
...Optional header arguments.
Returns:
Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().
API Stability:
Evolving.

int urlParse (Url *up, cchar *uri)

Parse a URL into its constituent components in the Url structure.

Parameters:
upURL object.
uriUri to parse.
Returns:
Zero if the uri parses successfully.
API Stability:
Evolving.

char * * urlPost (cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Issue a HTTP POST request.

Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
sizeSize of body data for request.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Headers arguments.
Returns:
Response body if successful, otherwise null. Caller must free.
API Stability:
Evolving.

Json * * urlPostJson (cchar *uri, cvoid *data, ssize len, cchar *headers, ...)

Issue a HTTP POST request and return parsed JSON.

Must only be called from a fiber.
Parameters:
uriHTTP URL to fetch.
dataBody data for request. Set to NULL if none.
lenSize of body data for request.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
Returns:
Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().
API Stability:
Evolving.

ssize urlRead (Url *up, char *buf, ssize bufsize)

Low level read routine to read response data for a request.

Description:
This routine may be called to progressively read response data. It should not be called if using urlGetResponse directly or indirectly via urlGet, urlPost, urlPostJson or urlJson. This routine will block the current fiber if necessary. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
bufBuffer to read into.
bufsizeSize of the buffer.
Returns:
The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.
API Stability:
Evolving.

void urlSetCerts (Url *up, cchar *ca, cchar *key, cchar *cert, cchar *revoke)

Define the certificates to use with TLS.

Parameters:
upURL object.
caCertificate authority to verify client certificates.
keyCertificate private key.
certCertificate text.
revokeCertificates to revoke.
API Stability:
Evolving.

void urlSetCiphers (Url *up, cchar *ciphers)

Set the list of available ciphers to use.

Parameters:
upURL object.
ciphersString list of available ciphers.
API Stability:
Prototype.

void urlSetDefaultRetries (int retries)

Set the default number of retries for requests.

Description:
This does not change the number of retries for existing Url objects.
Parameters:
retriesNumber of retries.
API Stability:
Evolving.

void urlSetDefaultTimeout (Ticks timeout)

Set the default request timeout to use for future URL instances.

Description:
This does not change the timeout for existing Url objects.
Parameters:
timeoutTimeout in milliseconds.
API Stability:
Evolving.

void urlSetError (Url *up, cchar *message, ...)

Set the URL internal error message.

Parameters:
upURL object.
messagePrintf style message format string.
...Message arguments.
API Stability:
Evolving.

void urlSetFlags (Url *up, int flags)

Set the URL flags.

Parameters:
upURL object.
flagsSet flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY.
API Stability:
Prototype.

void urlSetProtocol (Url *up, int protocol)

Set the HTTP protocol to use.

Parameters:
upURL object.
protocolSet to 0 for HTTP/1.0 or 1 for HTTP/1.1. Defaults to 1.
API Stability:
Prototype.

void urlSetStatusError (Url *up, int status, cchar *message, ...)

Set the URL internal error message and HTTP response status.

Parameters:
upURL object.
statusHTTP status code.
messagePrintf style message format string.
...Message arguments.
API Stability:
Evolving.

void urlSetTimeout (Url *up, Ticks timeout)

Set the request timeout to use for the specific URL object.

Parameters:
upURL object.
timeoutTimeout in milliseconds.
API Stability:
Evolving.

void urlSetVerify (Url *up, int verifyPeer, int verifyIssuer)

Control verification of TLS connections.

Parameters:
upURL object.
verifyPeerSet to true to verify the certificate of the remote peer.
verifyIssuerSet to true to verify the issuer of the peer certificate.
API Stability:
Evolving.

int urlStart (Url *up, cchar *method, cchar *uri, ssize txLen)

Start a Url request.

Description:
This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.
Must only be called from a fiber.
Parameters:
upURL object.
methodHTTP method verb.
uriHTTP URL to fetch.
txLenContent length of request body. This must match the length of data written with urlWrite. Set to -1 if the content length is not known.
Returns:
Zero if successful.
API Stability:
Evolving.

int urlUpload (Url *up, RList *files, RHash *forms, cchar *headers, ...)

Upload files in a request.

Description:
This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.
Must only be called from a fiber.
Parameters:
upURL object.
filesList of filenames to upload.
formsHash of key/value form values to add to the request.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Headers arguments.
Returns:
Zero if successful.
API Stability:
Evolving.

ssize urlWrite (Url *up, cvoid *data, ssize size)

Write body data for a request.

Description:
This routine will block the current fiber. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
dataBuffer of data to write.
sizeLength of data to write. Set to -1 to calculate the length of data as a null terminated string.
Returns:
The number of bytes actually written. On errors, returns a negative status code.
API Stability:
Evolving.

ssize urlWriteFile (Url *up, cchar *path)

Write a file contents for a request.

Description:
This routine will read the file contents and write to the client. It will block the current fiber. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
pathFile pathname.
Returns:
The number of bytes actually written. On errors, returns a negative status code.
API Stability:
Evolving.

ssize urlWriteFmt (Url *up, cchar *fmt, ...)

Write formatted body data for a request.

Description:
This routine will block the current fiber. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
fmtPrintf style format string.
...Format arguments.
Returns:
The number of bytes actually written. On errors, returns a negative status code.
API Stability:
Evolving.

int urlWriteHeaders (Url *up, cchar *headers, ...)

Write request headers.

Description:
This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will block the current fiber. Other fibers continue to run.
Must only be called from a fiber.
Parameters:
upURL object.
headersOptional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...Optional header arguments.
Returns:
Zero if successful.
API Stability:
Evolving.

Typedefs

typedef void(* UrlProc) (struct Url *up, int event, char *buf, ssize len).

URL callback procedure.

Parameters:
upURL object.
eventType of event. Set to URL_NOTIFY_IO, URL_NOTIFY_RETRY or URL_NOTIFY_DONE.
bufAssociated data.
lenAssociated data length.
API Stability:
Evolving.