4. APC rss aa module Client

4.1 Requirements for the client

The rss aa module client is the program which polls remote nodes for new items on a regular basis.

The rss aa module client must perform the following tasks:

• It must run on a regular basis.
• It must perform a database query for all active incoming feeds joined with the node table
• It must loop through these feeds. For every feed, it must
  • open an http connection to the remote node rss aa module server and make an http request. The request must contain all needed parameters: node user id, node password, user name, remote slice id, start timestamp (timestamp of the newest article that come through this feed), and optionally a list of category ids
  • receive and store the data sent by the remote http server
  • parse the rss aa module data
  • store new received items, categories and fields into the database.

4.2 Specification Detail

Running on a regular basis

The rss aa module client is implemented in a new file xmlclient.php3.

The program is expected to run on a regular basis. As a rule of thumb, we expect this interval to be 15 minutes. Because of the fact that the php scripts must be run through the http server, we need to implement the same mechanism as described in the APC Action Applications: E-Mail Subscription Service Specification, chapter 5, section 5.1 and 5.2 ( email-spec-5.html). The task described there is to run a program daily. the same mechanism should be used to run a program every 15 minutes.

Helper function: The XML parser

The XML parser should be put into an include file (name probably include/xmlparse.php3. It is used at two spots, from the rss aa module client specified in this chapter and from the administrative interface to establish a new incoming feed.

When a request has been made to a rss aa module server, the response is expected to be an xml document. PHP's xml functions are able to parse this result. The PHP xml functions are documented in the PHP 4 manual.

Using the PHP xml functions, a rss aa module parser function is created. Its name is apcrss_parse. The function takes a string containing the xml data to be parsed as an argument. It returns an array containing the result.

The parser function uses at least the PHP functions xml_parser_create, xml_set_element_handler, xml_set_character_data_handler, xml_parse and xml_parser_free.

The return value is an array containing all information that was in the xml data. The actual data structure within this array is not yet specified.

In cases of xml syntax errors, return the error message from the xml parser.

XML fetcher

In a new include file include/xml_fetch.php3, a function xml_fetch to fetch an xml data file through http is created. This function is used by the rss aa module client described in this chapter as well as by the administrative interface.

The function takes these arguments:

string url

The url to which the connection is to be made.

string parameters[]

Array containing the parameters that are to be posted with the http request.

The functions returns the received data in a string. On error, the function returns an empty string or, if available, an error message in the string which does not start with a '<' character.

The function

• opens an http connection to the url given as parameter url
• sends an http request with the parameters given in parameter parameters
• using a timeout of 15 seconds, reads the response from the remote node. If the timeout is exceeded, make a note in the log and return an error message
• closes the connection and returns the result.

The tasks

On execution, the script selects all incoming feeds from table external_feeds . The query must join

• with the nodes table on the fields external_feeds.node_name and nodes.name. This links the incoming feeds with feeding node information.
• The query must also join on table ef_categories on fields external_feeds.feed_id and ef_categories.feed_id. This links the incoming feed with its possible categories.
• Only those records from ef_categories that have a non empty string in the field ef_categories.target_category_id are needed because these are the categories that have been selected as a filter for the feed.

The program then loops through all feeds. All the following tasks are done for each feed respectively.

For each feed,

• use the xml_fetch function to fetch the xml data. Use the URL given in the field nodes.server_url and these parameters:

  node_username

  constant string ORG_NAME

  password

  string from field nodes.password

  user

  string from filed external_feeds.user_id

  slice_id

  string from field external_feeds.remote_slice_id

  start_timestamp

  numeric value from field external_feeds.txxxxx

  categories

  list of category ids which are selected (ef_categories.target_category_id is not empty). The list is composed by appending all category ids together, separated by a white space.

• check the first character of the xml data. If it is not a '<' (less than) character, consider this xml data to be an error message from the server. Log this error message and continue with the next feed.
• try to parse the response using the apcrss_parse function. If the function result indicates an xml error, log the error and continue with the next feed.
• Update the slice categories in the ef_categories table, that is, if the set of possible slice categories has changed, add new ones and remove those which have disappeared. When adding new categories, set the ef_categories.target_category_id to the id of the local slice's default category so the feed will now include the new categories. ef_categories.approved is set to 0.
• Update the field external_feeds .newest_item with the data received in the xml element newest_item_timestamp.
• Update the fields mapping from the remote slice to the local slice. If a new field appeared, map it automatically. If there is no matching field in the local slice, drop the new field. If an existing field vanished, delete the mapping.
• Update the items table. Only store items that have an id which is not already contained in the items database (avoid duplicates). Duplicate items would be identified by their item id. Also store the fields of the item. Do not store fields for which there is no field mapping. xml data field and database field mapping is done according to section APC AA RSS module definition: Element description . Set the category to the value of the field ef_categories .target_category_id. In the items table, set the field externally_fed to the remote node name and put the item into the correct bin according to ef_categories .approved. If more than one text format is contained in the items's text, select html if is is there, or plain text. Ignore other formats.