URL API

The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.

The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.

The library has configurable request/response timeouts and retries.

Features:

Supports transfer-chunk encoding for dynamic request bodies.
Control over headers
Full-duplex streaming support for minimal memory usage.
Full support for TLS including certificate verification.
Configurable retries and timeouts.
Parallelism via fiber coroutines.

The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.

Extensions

Url	Url request object.

Functions

Url *	urlAlloc(int flags)
	Allocate a URL object.
void	urlClose(Url *up)
	Close the underlying network socket associated with the URL object.
int	urlFetch(Url up, cchar method, cchar uri, cvoid data, ssize size, cchar *headers, ...)
	Fetch a URL.
ssize	urlFinalize(Url *url)
	Finalize request body.
void	urlFree(Url *up)
	Free a URL object.
char *	urlGet(cchar uri, cchar headers, ...)
	Get a URL using a HTTP GET request.
cchar *	urlGetError(Url *up)
	Get the URL internal error message.
cchar *	urlGetHeader(Url up, cchar header)
	Get a response HTTP header from the parsed response headers.
Json *	urlGetJson(cchar uri, cchar headers, ...)
	Issue a HTTP GET request and return parsed JSON.
Json *	urlGetJsonResponse(Url *up)
	Get the response to a URL request as a JSON object tree.
cchar *	urlGetResponse(Url *up)
	Get the response to a URL request as a string.
RBuf *	urlGetResponseBuf(Url *up)
	Get the response to a URL request in a buffer.
int	urlGetStatus(Url *up)
	Get the HTTP response status code for a request.
Json *	urlJson(Url up, cchar method, cchar uri, cvoid data, ssize size, cchar *headers, ...)
	Fetch a URL and return a JSON response.
int	urlParse(Url up, cchar uri)
	Parse a URL into its constituent components in the Url structure.
char *	urlPost(cchar uri, cvoid data, ssize size, cchar *headers, ...)
	Issue a HTTP POST request.
Json *	urlPostJson(cchar uri, cvoid data, ssize len, cchar *headers, ...)
	Issue a HTTP POST request and return parsed JSON.
ssize	urlRead(Url up, char buf, ssize bufsize)
	Low level read routine to read response data for a request.
void	urlSetCerts(Url up, cchar ca, cchar key, cchar cert, cchar *revoke)
	Define the certificates to use with TLS.
void	urlSetCiphers(Url up, cchar ciphers)
	Set the list of available ciphers to use.
void	urlSetDefaultRetries(int retries)
	Set the default number of retries for requests.
void	urlSetDefaultTimeout(int timeout)
	Set the default request timeout to use for future URL instances.
void	urlSetError(Url up, cchar message, ...)
	Set the URL internal error message.
void	urlSetFlags(Url *up, int flags)
	Set the URL flags.
void	urlSetProtocol(Url *up, int protocol)
	Set the HTTP protocol to use.
void	urlSetStatusError(Url up, int status, cchar message, ...)
	Set the URL internal error message and HTTP response status.
void	urlSetTimeout(Url *up, int timeout)
	Set the request timeout to use for the specific URL object.
void	urlSetVerify(Url *up, int verifyPeer, int verifyIssuer)
	Control verification of TLS connections.
int	urlStart(Url up, cchar method, cchar *uri, ssize txLen)
	Start a Url request.
int	urlUpload(Url up, RList files, RHash forms, cchar headers, ...)
	Upload files in a request.
ssize	urlWrite(Url up, cvoid data, ssize size)
	Write body data for a request.
ssize	urlWriteFile(Url up, cchar path)
	Write a file contents for a request.
ssize	urlWriteFmt(Url up, cchar fmt, ...)
	Write formatted body data for a request.
int	urlWriteHeaders(Url up, cchar headers, ...)
	Write request headers.

Typedefs

UrlProc

URL callback procedure.

Defines

#define	URL_CODE_ACCEPTED 202
	The request has been accepted and processing is continuing.
#define	URL_CODE_BAD_GATEWAY 502
	The server cannot act as a gateway for the given request.
#define	URL_CODE_BAD_METHOD 405
	The request HTTP method was not supported by the resource.
#define	URL_CODE_BAD_REQUEST 400
	The request is malformed.
#define	URL_CODE_BAD_VERSION 505
	The server does not support the HTTP protocol version.
#define	URL_CODE_CONFLICT 409
	The request had a conflict in the request headers and URI.
#define	URL_CODE_CONTINUE 100
	Continue with request, only parial content transmitted.
#define	URL_CODE_CREATED 201
	The request has completed and a new resource was created.
#define	URL_CODE_EXPECTATION_FAILED 417
	The server cannot satisfy the Expect header requirements.
#define	URL_CODE_FORBIDDEN 403
	The request was legal, but the server refuses to process.
#define	URL_CODE_GATEWAY_TIMEOUT 504
	The server gateway timed out waiting for the upstream server.
#define	URL_CODE_GONE 410
	The requested resource is no longer available.
#define	URL_CODE_IM_A_TEAPOT 418
	Short and stout error code (RFC 2324).
#define	URL_CODE_INSUFFICIENT_STORAGE 507
	The server has insufficient storage to complete the request.
#define	URL_CODE_INTERNAL_SERVER_ERROR 500
	Server processing or configuration error.
#define	URL_CODE_LENGTH_REQUIRED 411
	The request did not specify a required content length.
#define	URL_CODE_MOVED_PERMANENTLY 301
	The requested URI has moved permanently to a new location.
#define	URL_CODE_MOVED_TEMPORARILY 302
	The URI has moved temporarily to a new location.
#define	URL_CODE_NO_CONTENT 204
	The request has completed and there is no response to send.
#define	URL_CODE_NO_RESPONSE 444
	The connection was closed with no response to the client.
#define	URL_CODE_NOT_ACCEPTABLE 406
	The requested resource cannot generate the required content.
#define	URL_CODE_NOT_AUTHORITATIVE 203
	The request has completed but content may be from another source.
#define	URL_CODE_NOT_FOUND 404
	The requested resource was not found.
#define	URL_CODE_NOT_IMPLEMENTED 501
	The server does not recognize the request or method.
#define	URL_CODE_NOT_MODIFIED 304
	The requested resource has changed since the last request.
#define	URL_CODE_OK 200
	The request completed successfully.
#define	URL_CODE_PARTIAL 206
	The request has completed and is returning parial content.
#define	URL_CODE_PAYMENT_REQUIRED 402
	Reserved for future use.
#define	URL_CODE_PRECOND_FAILED 412
	The server cannot satisfy one of the request preconditions.
#define	URL_CODE_RANGE_NOT_SATISFIABLE 416
	The request content range does not exist for the resource.
#define	URL_CODE_REQUEST_TIMEOUT 408
	The server timed out waiting for the request to complete.
#define	URL_CODE_REQUEST_TOO_LARGE 413
	The request is too large for the server to process.
#define	URL_CODE_REQUEST_URL_TOO_LARGE 414
	The request URI is too long for the server to process.
#define	URL_CODE_RESET 205
	The request has completed with no content.
#define	URL_CODE_SEE_OTHER 303
	The requested URI can be found at another URI location.
#define	URL_CODE_SERVICE_UNAVAILABLE 503
	The server is currently unavailable or overloaded.
#define	URL_CODE_SWITCHING 101
	Switching protocols.
#define	URL_CODE_TEMPORARY_REDIRECT 307
	The request should be repeated at another URI location.
#define	URL_CODE_UNAUTHORIZED 401
	Authentication for the request has failed.
#define	URL_CODE_UNSUPPORTED_MEDIA_TYPE 415
	The request media type is not supported by the server or resource.
#define	URL_CODE_USE_PROXY 305
	The requested resource must be accessed via the location proxy.
#define	URL_HTTP_0 0x20
	Use HTTP/1.0.
#define	URL_NOTIFY_DONE 3
	Request is complete.
#define	URL_NOTIFY_IO 1
	Streaming I/O event.
#define	URL_NOTIFY_RETRY 2
	Retry initiated.
#define	URL_SHOW_NONE 0x1
	Trace nothing.
#define	URL_SHOW_REQ_BODY 0x2
	Trace request body.
#define	URL_SHOW_REQ_HEADERS 0x4
	Trace request headers.
#define	URL_SHOW_RESP_BODY 0x8
	Trace response body.
#define	URL_SHOW_RESP_HEADERS 0x10
	Trace response headers.

Url

Url request object.

Description:: The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.

API Stability:: Evolving.

Fields:

char *	boundary	Multipart mime upload file boundary.
uint	certsDefined	Certificates have been defined.
uint	chunked	Request is using transfer chunk encoding.
uint	close	Connection should be closed on completion of the current request.
Ticks	deadline	Request time limit expiry deadline.
char *	error	Error message.
uint	flags	Alloc flags.
uint	gotResponse	Response has been read.
cchar *	hash	Request hash portion.
cchar *	host	Request host.
cchar *	path	Request path without leading "/" and query/ref.
int	port	Request port.
uint	protocol	Use HTTP/1.0 without keep-alive. Defaults to HTTP/1.1.
cchar *	query	Request query portion.
cchar *	redirect	Redirect location.
char *	response	Response as a string.
RBuf *	responseBuf	Buffer to hold complete response.
uint	retries	Client implemented retry field.
RBuf *	rx	Buffer for progressive reading response data.
RBuf *	rxHeaders	Buffer for Rx headers.
ssize	rxLen	Length of rx body.
ssize	rxRemaining	Remaining rx data to read.
cchar *	scheme	Request scheme.
RSocket *	sock	Network socket.
uint	status	Response (rx) status.
Ticks	timeout	Request timeout.
ssize	txLen	Length of tx body.
char *	urlbuf	HTTP request method. Parsed and tokenized url.
uint	wroteHeaders	Tx headers have been written.

Url * * urlAlloc (int flags)

Allocate a URL object.

Description:: A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.

Parameters:

flags

Set flags to URL_SHOW_REQ_HEADERS | URL_SHOW_REQ_BODY | URL_SHOW_RESP_HEADERS | URL_SHOW_RESP_BODY. Defaults to 0.

Returns:: The url object.

API Stability:: Evolving.

void urlClose (Url *up)

Close the underlying network socket associated with the URL object.

Description:: This is not normally necessary to invoke unless you want to force the socket connection to be recreated before issuing a subsequent request.

Parameters:

up	URL object.

API Stability:: Evolving.

int urlFetch (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL.

Description:: This routine issues a HTTP request with optional body data and returns the HTTP status code. This routine will return before the response body data has been read. Use $urlRead or $urlGetResponse to read the response body data. This routine will block the current fiber while waiting for the request to respond. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
method	HTTP method verb.
uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
size	Size of body data for request. Set to 0 if none.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Optional header arguments.

Returns:: Response HTTP status code. Use urlGetResponse or urlRead to read the response.

API Stability:: Evolving.

ssize urlFinalize (Url *url)

Finalize request body.

Description:: This routine will call urlWrite(web, NULL, 0);.

Parameters:

url	Url object.

Returns:: Number of bytes written. Returns a negative status code on errors.

API Stability:: Evolving.

void urlFree (Url *up)

Free a URL object.

Parameters:

up	URL object.

API Stability:: Evolving.

char * * urlGet (cchar *uri, cchar *headers, ...)

Get a URL using a HTTP GET request.

Description:: This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Optional header arguments.

Returns:: Response body if successful, otherwise null. Caller must free.

API Stability:: Evolving.

cchar * * urlGetError (Url *up)

Get the URL internal error message.

Description:: Errors are defined for unexpected errors. HTTP requests that respond with a non-200 status do not set the error message.

Parameters:

up	URL object.

Returns:: The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.

API Stability:: Evolving.

cchar * * urlGetHeader (Url *up, cchar *header)

Get a response HTTP header from the parsed response headers.

Parameters:

up	URL object.
header	HTTP header name. This can be any case. For example: "Authorization".

Returns:: The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.

API Stability:: Evolving.

Json * * urlGetJson (cchar *uri, cchar *headers, ...)

Issue a HTTP GET request and return parsed JSON.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Optional header arguments.

Returns:: Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().

API Stability:: Evolving.

Json * * urlGetJsonResponse (Url *up)

Get the response to a URL request as a JSON object tree.

Description:: After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.

Returns:: The response body as parsed JSON object. Caller must free the result via jsonFree.

API Stability:: Evolving.

cchar * * urlGetResponse (Url *up)

Get the response to a URL request as a string.

Description:: After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

If receiving a binary response, use urlGetResponseBuf instead.

: Must only be called from a fiber.

Parameters:

up	URL object.

Returns:: The response body as a string. Caller must NOT free.

API Stability:: Evolving.

RBuf * * urlGetResponseBuf (Url *up)

Get the response to a URL request in a buffer.

Description:: After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.

Returns:: The response body as runtime buffer. Caller must NOT free.

API Stability:: Evolving.

int urlGetStatus (Url *up)

Get the HTTP response status code for a request.

Parameters:

up	URL object.

Returns:: The HTTP status code for the most recently completed request.

API Stability:: Evolving.

Json * * urlJson (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL and return a JSON response.

Description:: This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
method	HTTP method verb.
uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
size	Size of body data for request. Set to 0 if none.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used.
...	Optional header arguments.

Returns:: Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().

API Stability:: Evolving.

int urlParse (Url *up, cchar *uri)

Parse a URL into its constituent components in the Url structure.

Parameters:

up	URL object.
uri	Uri to parse.

Returns:: Zero if the uri parses successfully.

API Stability:: Evolving.

char * * urlPost (cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Issue a HTTP POST request.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
size	Size of body data for request.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Headers arguments.

Returns:: Response body if successful, otherwise null. Caller must free.

API Stability:: Evolving.

Json * * urlPostJson (cchar *uri, cvoid *data, ssize len, cchar *headers, ...)

Issue a HTTP POST request and return parsed JSON.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
len	Size of body data for request.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".

Returns:: Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().

API Stability:: Evolving.

ssize urlRead (Url *up, char *buf, ssize bufsize)

Low level read routine to read response data for a request.

Description:: This routine may be called to progressively read response data. It should not be called if using urlGetResponse directly or indirectly via urlGet, urlPost, urlPostJson or urlJson. This routine will block the current fiber if necessary. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
buf	Buffer to read into.
bufsize	Size of the buffer.

Returns:: The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.

API Stability:: Evolving.

void urlSetCerts (Url *up, cchar *ca, cchar *key, cchar *cert, cchar *revoke)

Define the certificates to use with TLS.

Parameters:

up	URL object.
ca	Certificate authority to verify client certificates.
key	Certificate private key.
cert	Certificate text.
revoke	Certificates to revoke.

API Stability:: Evolving.

void urlSetCiphers (Url *up, cchar *ciphers)

Set the list of available ciphers to use.

Parameters:

up	URL object.
ciphers	String list of available ciphers.

API Stability:: Prototype.

void urlSetDefaultRetries (int retries)

Set the default number of retries for requests.

Description:: This does not change the number of retries for existing Url objects.

Parameters:

retries

Number of retries.

API Stability:: Evolving.

void urlSetDefaultTimeout (int timeout)

Set the default request timeout to use for future URL instances.

Description:: This does not change the timeout for existing Url objects.

Parameters:

timeout

Timeout in milliseconds.

API Stability:: Evolving.

void urlSetError (Url *up, cchar *message, ...)

Set the URL internal error message.

Parameters:

up	URL object.
message	Printf style message format string.
...	Message arguments.

API Stability:: Evolving.

void urlSetFlags (Url *up, int flags)

Set the URL flags.

Parameters:

up	URL object.
flags	Set flags to URL_SHOW_REQ_HEADERS \| URL_SHOW_REQ_BODY \| URL_SHOW_RESP_HEADERS \| URL_SHOW_RESP_BODY.

API Stability:: Prototype.

void urlSetProtocol (Url *up, int protocol)

Set the HTTP protocol to use.

Parameters:

up	URL object.
protocol	Set to 0 for HTTP/1.0 or 1 for HTTP/1.1. Defaults to 1.

API Stability:: Prototype.

void urlSetStatusError (Url *up, int status, cchar *message, ...)

Set the URL internal error message and HTTP response status.

Parameters:

up	URL object.
status	HTTP status code.
message	Printf style message format string.
...	Message arguments.

API Stability:: Evolving.

void urlSetTimeout (Url *up, int timeout)

Set the request timeout to use for the specific URL object.

Parameters:

up	URL object.
timeout	Timeout in milliseconds.

API Stability:: Evolving.

void urlSetVerify (Url *up, int verifyPeer, int verifyIssuer)

Control verification of TLS connections.

Parameters:

up	URL object.
verifyPeer	Set to true to verify the certificate of the remote peer.
verifyIssuer	Set to true to verify the issuer of the peer certificate.

API Stability:: Evolving.

int urlStart (Url *up, cchar *method, cchar *uri, ssize txLen)

Start a Url request.

Description:: This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.

: Must only be called from a fiber.

Parameters:

up	URL object.
method	HTTP method verb.
uri	HTTP URL to fetch.
txLen	Content length of request body. This must match the length of data written with urlWrite. Set to -1 if the content length is not known.

Returns:: Zero if successful.

API Stability:: Evolving.

int urlUpload (Url *up, RList *files, RHash *forms, cchar *headers, ...)

Upload files in a request.

Description:: This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.

: Must only be called from a fiber.

Parameters:

up	URL object.
files	List of filenames to upload.
forms	Hash of key/value form values to add to the request.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Headers arguments.

Returns:: Zero if successful.

API Stability:: Evolving.

ssize urlWrite (Url *up, cvoid *data, ssize size)

Write body data for a request.

Description:: This routine will block the current fiber. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
data	Buffer of data to write.
size	Length of data to write. Set to -1 to calculate the length of data as a null terminated string.

Returns:: The number of bytes actually written. On errors, returns a negative status code.

API Stability:: Evolving.

ssize urlWriteFile (Url *up, cchar *path)

Write a file contents for a request.

Description:: This routine will read the file contents and write to the client. It will block the current fiber. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
path	File pathname.

Returns:: The number of bytes actually written. On errors, returns a negative status code.

API Stability:: Evolving.

ssize urlWriteFmt (Url *up, cchar *fmt, ...)

Write formatted body data for a request.

Description:: This routine will block the current fiber. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
fmt	Printf style format string.
...	Format arguments.

Returns:: The number of bytes actually written. On errors, returns a negative status code.

API Stability:: Evolving.

int urlWriteHeaders (Url *up, cchar *headers, ...)

Write request headers.

Description:: This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will block the current fiber. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Optional header arguments.

Returns:: Zero if successful.

API Stability:: Evolving.

Typedefs

typedef void(* UrlProc) (struct Url *up, int event, char *buf, ssize len).

URL callback procedure.

Parameters:

up	URL object.
event	Type of event. Set to URL_NOTIFY_IO, URL_NOTIFY_RETRY or URL_NOTIFY_DONE.
buf	Associated data.
len	Associated data length.

API Stability:: Evolving.