URL API

The URL library is a fast, minimal embedded HTTP Client. It is ideal for issuing REST HTTP requests and supports parallel execution via fiber coroutines.

The URL library is a streaming client and supports full-duplex for request body and response content. It transparently manages transfer chunk encoding for requests and responses.

The library has configurable request/response timeouts and retries.

Features:

Supports transfer-chunk encoding for dynamic request bodies.
Control over headers
Full-duplex streaming support for minimal memory usage.
Full support for TLS including certificate verification.
Configurable retries and timeouts.
Parallelism via fiber coroutines.

The library does not support basic or digest authentication as there are unresolved security issues with these mechanisms.

Extensions

Url	Url request object.

Functions

Url*	urlAlloc(void)
	Allocate a URL object.
void	urlClose(Url *up)
	Close the underlying network socket associated with the URL object.
int	urlFetch(Url up, cchar method, cchar uri, cvoid data, ssize size, cchar *headers, ...)
	Fetch a URL.
Json*	urlFetchJson(Url up, cchar method, cchar uri, cvoid data, ssize size, cchar *headers, ...)
	Fetch a URL and return a JSON response.
void	urlFree(Url *up)
	Free a URL object.
char*	urlGet(cchar uri, cchar headers, ...)
	Get a URL using a HTTP GET request.
cchar*	urlGetError(Url *up)
	Get the URL internal error message.
cchar*	urlGetHeader(Url up, cchar header)
	Get a response HTTP header.
Json*	urlGetJsonResponse(Url *up)
	Get the response to a URL request as a JSON object tree.
cchar*	urlGetResponse(Url *up)
	Get the response to a URL request.
int	urlGetStatus(Url *up)
	Get the HTTP response status code for a request.
int	urlParse(Url up, cchar uri)
	Parse a URL into its constituent components in the Url structure.
char*	urlPost(cchar uri, cvoid data, ssize size, cchar *headers, ...)
	Issue a HTTP POST request.
Json*	urlPostJson(cchar uri, cvoid data, ssize len, cchar *headers, ...)
	Issue a HTTP POST request and return parsed JSON.
ssize	urlRead(Url up, char buf, ssize bufsize)
	Read response data for a request.
void	urlSetCerts(Url up, cchar key, cchar cert, cchar ca, cchar *revoke)
	Define the certificates to use with TLS.
void	urlSetDefaultRetries(int retries)
	Set the default number of retries for requests.
void	urlSetDefaultTimeout(int timeout)
	Set the default request timeout to use for future URL instances.
void	urlSetError(Url up, cchar message, ...)
	Set the URL internal error message.
void	urlSetTimeout(Url *up, int timeout)
	Set the request timeout to use for the specific URL object.
void	urlSetVerify(Url *up, int verifyPeer, int verifyIssuer)
	Control verification of TLS connections.
int	urlStart(Url up, cchar method, cchar *uri, ssize txLen)
	Start a Url request.
ssize	urlWrite(Url up, cvoid data, ssize size)
	Write body data for a request.
int	urlWriteHeaders(Url up, cchar headers, ...)
	Write request headers.

Typedefs

UrlProc

URL callback procedure.

Defines

#define	URL_CODE_ACCEPTED 202
	The request has been accepted and processing is continuing.
#define	URL_CODE_BAD_GATEWAY 502
	The server cannot act as a gateway for the given request.
#define	URL_CODE_BAD_METHOD 405
	The request HTTP method was not supported by the resource.
#define	URL_CODE_BAD_REQUEST 400
	The request is malformed.
#define	URL_CODE_BAD_VERSION 505
	The server does not support the HTTP protocol version.
#define	URL_CODE_CONFLICT 409
	The request had a conflict in the request headers and URI.
#define	URL_CODE_CONTINUE 100
	Continue with request, only parial content transmitted.
#define	URL_CODE_CREATED 201
	The request has completed and a new resource was created.
#define	URL_CODE_EXPECTATION_FAILED 417
	The server cannot satisfy the Expect header requirements.
#define	URL_CODE_FORBIDDEN 403
	The request was legal, but the server refuses to process.
#define	URL_CODE_GATEWAY_TIMEOUT 504
	The server gateway timed out waiting for the upstream server.
#define	URL_CODE_GONE 410
	The requested resource is no longer available.
#define	URL_CODE_IM_A_TEAPOT 418
	Short and stout error code (RFC 2324).
#define	URL_CODE_INSUFFICIENT_STORAGE 507
	The server has insufficient storage to complete the request.
#define	URL_CODE_INTERNAL_SERVER_ERROR 500
	Server processing or configuration error.
#define	URL_CODE_LENGTH_REQUIRED 411
	The request did not specify a required content length.
#define	URL_CODE_MOVED_PERMANENTLY 301
	The requested URI has moved permanently to a new location.
#define	URL_CODE_MOVED_TEMPORARILY 302
	The URI has moved temporarily to a new location.
#define	URL_CODE_NO_CONTENT 204
	The request has completed and there is no response to send.
#define	URL_CODE_NO_RESPONSE 444
	The connection was closed with no response to the client.
#define	URL_CODE_NOT_ACCEPTABLE 406
	The requested resource cannot generate the required content.
#define	URL_CODE_NOT_AUTHORITATIVE 203
	The request has completed but content may be from another source.
#define	URL_CODE_NOT_FOUND 404
	The requested resource was not found.
#define	URL_CODE_NOT_IMPLEMENTED 501
	The server does not recognize the request or method.
#define	URL_CODE_NOT_MODIFIED 304
	The requested resource has changed since the last request.
#define	URL_CODE_OK 200
	The request completed successfully.
#define	URL_CODE_PARTIAL 206
	The request has completed and is returning parial content.
#define	URL_CODE_PAYMENT_REQUIRED 402
	Reserved for future use.
#define	URL_CODE_PRECOND_FAILED 412
	The server cannot satisfy one of the request preconditions.
#define	URL_CODE_RANGE_NOT_SATISFIABLE 416
	The request content range does not exist for the resource.
#define	URL_CODE_REQUEST_TIMEOUT 408
	The server timed out waiting for the request to complete.
#define	URL_CODE_REQUEST_TOO_LARGE 413
	The request is too large for the server to process.
#define	URL_CODE_REQUEST_URL_TOO_LARGE 414
	The request URI is too long for the server to process.
#define	URL_CODE_RESET 205
	The request has completed with no content.
#define	URL_CODE_SEE_OTHER 303
	The requested URI can be found at another URI location.
#define	URL_CODE_SERVICE_UNAVAILABLE 503
	The server is currently unavailable or overloaded.
#define	URL_CODE_SWITCHING 101
	Switching protocols.
#define	URL_CODE_TEMPORARY_REDIRECT 307
	The request should be repeated at another URI location.
#define	URL_CODE_UNAUTHORIZED 401
	Authentication for the request has failed.
#define	URL_CODE_UNSUPPORTED_MEDIA_TYPE 415
	The request media type is not supported by the server or resource.
#define	URL_CODE_USE_PROXY 305
	The requested resource must be accessed via the location proxy.
#define	URL_NOTIFY_DONE 3
	Request is complete.
#define	URL_NOTIFY_IO 1
	Streaming I/O event.
#define	URL_NOTIFY_RETRY 2
	Retry initiated.

Url

Url request object.

Description:: The Url service is a streaming HTTP request client. This service requires the use of fibers from the portable runtime.

API Stability:: Evolving.

Fields:

uint	certsDefined	Certificates have been defined.
uint	chunked	Request is using transfer chunk encoding.
uint	close	Connection should be closed on completion of the current request.
Ticks	deadline	Request time limit expiry deadline.
char *	error	Error message.
cchar *	host	Request host.
cchar *	path	Request path without leading "/" and query/ref.
int	port	Request port.
cchar *	query	Request query portion.
cchar *	reference	Request reference portion.
RBuf *	response	Buffer to hold complete response.
uint	responseRead	Response has been read.
uint	retries	Client implemented retry field.
RBuf *	rx	Buffer for reading responses.
char *	rxHeaders	Rx headers.
ssize	rxLen	Length of rx body.
ssize	rxRemaining	Rx headers.
cchar *	scheme	Request scheme.
RSocket *	sock	Network socket.
uint	status	Response (rx) status.
Ticks	timeout	Request timeout.
ssize	txLen	Length of tx body.
char *	urlbuf	HTTP request method. Parsed and tokenized url.
uint	wroteHeaders	Tx headers have been written.

Url * urlAlloc (void )

Allocate a URL object.

Description:: A URL object represents a network connection on which one or more HTTP client requests may be issued one at a time.

Returns:: The url object.

API Stability:: Evolving.

See Also:

void urlClose (Url *up)

Close the underlying network socket associated with the URL object.

Parameters:

up	URL object.

API Stability:: Evolving.

See Also:

int urlFetch (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL.

Description:: This routine issues a HTTP request and reads and buffers the response. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
method	HTTP method verb.
uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
size	Size of body data for request. Set to 0 if none.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Optional header arguments.

Returns:: Response HTTP status code. Use urlGetResponse or urlRead to read the response.

API Stability:: Evolving.

See Also:

Json * urlFetchJson (Url *up, cchar *method, cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Fetch a URL and return a JSON response.

Description:: This routine issues a HTTP request and reads and parses the response into a JSON object tree. This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
method	HTTP method verb.
uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
size	Size of body data for request. Set to 0 if none.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n". If the headers are not provided, a headers value of "Content-Type: application/json\r\n" is used.
...	Optional header arguments.

Returns:: Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Use urlGetError to get any error string and urlGetStatus to get the HTTP status code. Caller must free via jsonFree().

API Stability:: Evolving.

See Also:

void urlFree (Url *up)

Free a URL object.

Parameters:

up	URL object.

API Stability:: Evolving.

See Also:

char * urlGet (cchar *uri, cchar *headers, ...)

Get a URL using a HTTP GET request.

Description:: This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".

Returns:: Response body if successful, otherwise null. Caller must free.

API Stability:: Evolving.

See Also:

cchar * urlGetError (Url *up)

Get the URL internal error message.

Parameters:

up	URL object.

Returns:: The URL error message for the most recent request. Returns NULL if no error message defined. Caller must NOT free this message.

API Stability:: Evolving.

See Also:

cchar * urlGetHeader (Url *up, cchar *header)

Get a response HTTP header.

Parameters:

up	URL object.
header	HTTP header name. This can be any case. For example: "Authorization".

Returns:: The value of the HTTP header. Returns NULL if not defined. Caller must NOT free the returned string.

API Stability:: Evolving.

See Also:

Json * urlGetJsonResponse (Url *up)

Get the response to a URL request as a JSON object tree.

Description:: After issuing urlFetch, urlGet or urlPost, this routine may be called to read and parse the response as a JSON object. This call should only be used when the response is a valid JSON UTF-8 string. This routine buffers the entire response body and creates the parsed JSON tree.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.

Returns:: The response body as parsed JSON object. Caller must free the result via jsonFree.

API Stability:: Evolving.

See Also:

cchar * urlGetResponse (Url *up)

Get the response to a URL request.

Description:: After issuing urlFetch, urlGet or urlPost, this routine may be called to read, buffer and return the response body. This call should only be used when the response is a valid UTF-8 string. Otherwise, use urlRead to read the response body. As this routine buffers the entire response body, it should only be used for relatively small requests. Otherwise, the memory footprint of the application may be larger than desired.

This routine will block the current fiber while waiting for the request to complete. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.

Returns:: The response body as a string. Caller must NOT free.

API Stability:: Evolving.

See Also:

int urlGetStatus (Url *up)

Get the HTTP response status code for a request.

Parameters:

up	URL object.

Returns:: The HTTP status code for the most recently completed request.

API Stability:: Evolving.

See Also:

int urlParse (Url *up, cchar *uri)

Parse a URL into its constituent components in the Url structure.

Parameters:

up	URL object.
uri	Uri to parse.

Returns:: Zero if the uri parses successfully.

API Stability:: Evolving.

See Also:

char * urlPost (cchar *uri, cvoid *data, ssize size, cchar *headers, ...)

Issue a HTTP POST request.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
size	Size of body data for request.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".

Returns:: Response body if successful, otherwise null. Caller must free.

API Stability:: Evolving.

See Also:

Json * urlPostJson (cchar *uri, cvoid *data, ssize len, cchar *headers, ...)

Issue a HTTP POST request and return parsed JSON.

: Must only be called from a fiber.

Parameters:

uri	HTTP URL to fetch.
data	Body data for request. Set to NULL if none.
len	Size of body data for request.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".

Returns:: Parsed JSON response. If the request does not return a HTTP 200 status code or the response is not JSON, the request returns NULL. Caller must free via jsonFree().

API Stability:: Evolving.

See Also:

ssize urlRead (Url *up, char *buf, ssize bufsize)

Read response data for a request.

Description:: This routine will block the current fiber if necessary. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
buf	Buffer to read into.
bufsize	Size of the buffer.

Returns:: The number of bytes read. Returns < 0 on errors. Returns 0 when there is no more data to read.

API Stability:: Evolving.

See Also:

void urlSetCerts (Url *up, cchar *key, cchar *cert, cchar *ca, cchar *revoke)

Define the certificates to use with TLS.

Parameters:

up	URL object.
key	Certificate private key.
cert	Certificate text.
ca	Certificate authority text.
revoke	Certificates to revoke.

API Stability:: Evolving.

See Also:

void urlSetDefaultRetries (int retries)

Set the default number of retries for requests.

Description:: This does not change the number of retries for existing Url objects.

Parameters:

retries

Number of retries.

API Stability:: Evolving.

See Also:

void urlSetDefaultTimeout (int timeout)

Set the default request timeout to use for future URL instances.

Description:: This does not change the timeout for existing Url objects.

Parameters:

timeout

Timeout in milliseconds.

API Stability:: Evolving.

See Also:

void urlSetError (Url *up, cchar *message, ...)

Set the URL internal error message.

Parameters:

up	URL object.
message	Printf style message format string.
...	Message arguments.

API Stability:: Evolving.

See Also:

void urlSetTimeout (Url *up, int timeout)

Set the request timeout to use for the specific URL object.

Parameters:

up	URL object.
timeout	Timeout in milliseconds.

API Stability:: Evolving.

See Also:

void urlSetVerify (Url *up, int verifyPeer, int verifyIssuer)

Control verification of TLS connections.

Parameters:

up	URL object.
verifyPeer	Set to true to verify the certificate of the remote peer.
verifyIssuer	Set to true to verify the issuer of the peer certificate.

API Stability:: Evolving.

See Also:

int urlStart (Url *up, cchar *method, cchar *uri, ssize txLen)

Start a Url request.

Description:: This is a low level API that initiates a connection to a remote HTTP resource. Use urlWriteHeaders and urlWrite to send the request.

: Must only be called from a fiber.

Parameters:

up	URL object.
method	HTTP method verb.
uri	HTTP URL to fetch.
txLen	Content length of request body. This must match the length of data written with urlWrite. Set to -1 if the content length is not known.

Returns:: Zero if successful.

API Stability:: Evolving.

See Also:

ssize urlWrite (Url *up, cvoid *data, ssize size)

Write body data for a request.

Description:: This routine will block the current fiber. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
data	Buffer of data to write.
size	Length of data to write.

Returns:: The number of bytes actually written.

API Stability:: Evolving.

See Also:

int urlWriteHeaders (Url *up, cchar *headers, ...)

Write request headers.

Description:: This will write the HTTP request line and supplied headers. If Host and/or Content-Length are not supplied, they will be added if known. This routine will block the current fiber. Other fibers continue to run.

: Must only be called from a fiber.

Parameters:

up	URL object.
headers	Optional request headers. This parameter is a printf style formatted pattern with following arguments. Individual header lines must be terminated with "\r\n".
...	Optional header arguments.

Returns:: Zero if successful.

API Stability:: Evolving.

See Also:

Typedefs

typedef void(* UrlProc) (struct Url *up, int event, char *buf, ssize len).

URL callback procedure.

Parameters:

up	URL object.
event	Type of event. Set to URL_NOTIFY_IO, URL_NOTIFY_RETRY or URL_NOTIFY_DONE.
buf	Associated data.
len	Associated data length.

API Stability:: Evolving.

See Also: