Ab Initio Component | NORMALIZE : Part 1

Purpose

•  NORMALIZE is used to generate multiple output records from each of its input records. You can directly specify the number of output records for each input record, or you can make the number of output records dependent on a calculation. 

• In contrast, to consolidate groups of related records into a single record with a vector field for each group — the inverse of NORMALIZE — you can use the accumulation function of the ROLLUP component.

 Parameters for NORMALIZE

Note : Only the important parameter are covered the uncovered parameters are the common parameters which is been covered in other components details

transform (filename or string, required)

• The name of the file containing the types and transform functions, or a transform string.

error_group (string, optional)

• This parameter defines name of the error group to which this component belongs. The component sends its error output to the HANDLE ERRORS component with a matching error_group value.

log_group (string, optional)

• This parameter defines name of the log group to which this component belongs. The component sends its log output to the HANDLE LOGS component with a matching log_group value.

reject-threshold (choice, optional)

•  The component’s tolerance for reject events
• This parameter defines after how many reject records the component processing is aborted.
  

 
log_intermediate (choice, optional)

• This parameter defines how often the component sends an intermediate record to its log port. Available only when the logging parameter is set to True.

• For example, if you select 50, the component sends every 50th intermediate record to its log port.

Transform package for NORMALIZE

• Mentioned below describe the functions, types, and variables available in the transform package for NORMALIZE:

•     Simple transform
•     Multistage transform
•     How many records to produce
•     Other optional transform functions

 

Ab Initio Component | BROADCAST

 Purpose of BROADCAST

• BROADCAST component combines in an arbitrary order all records it receives into a single flow and writes a copy of that flow to each of its output flow partitions.

• You can use BROADCAST to increase data parallelism when you have connected a single fan-out flow to the out port, or to increase component parallelism when you have connected multiple straight flows to the out port.

Parameters for BROADCAST

• Broadcast components has no parameter

Runtime behavior of BROADCAST

BROADCAST does the following:

    1. It reads records from all flows on the in port.

    2. Then combines the records in an arbitrary order into a single flow.

    3. Then copies all the records to all the flow partitions connected to the out port.

Ab Initio Component | REFORMAT: Part 3

  ..Continue from Part 2..

 

 Runtime behavior of REFORMAT

• First it read the port count(n) from count parameter of component the n in outn gives each out port a unique number. Each outn port has a corresponding rejectn and errorn port. REFORMAT does the following:
  

1. REFORMAT reads a record from the in port of component. 
2.  If the select parameter has an expression specified, It uses the expression to evaluate the input record: 

•  if the expression evaluates to false (0), then REFORMAT discards the input record and starts over with step 1.

•  If the expression produces NULL, then REFORMAT writes a descriptive error message and stops execution of the graph.

• If the expression evaluates to true (anything other than 0 or NULL), then REFORMAT begins processing the input record.

3. If the select parameter does not have a value, REFORMAT begins processing the input record.

 

4.  REFORMAT determines whether a transform function is specified in either the output-index or output-indexes parameter:

• If neither output-index nor output-indexes has a value (the usual case when there is only one out port), REFORMAT sends the input record to every transform-out port pair, beginning with out0 and progressing sequentially.

• If output-index or output-indexes has a value, REFORMAT evaluates the specified index transform. If output-indexes is defined, it should return a vector of port index values. If output-index is defined, it should return a single port index value.
  

•  REFORMAT uses one or more values from the index transform to determine the appropriate transform-output port pair or pairs for the input record. If the index transform returns more than one value, REFORMAT sends the record to each of the appropriate ports, starting with the lowest numbered port and progressing to the other ports sequentially.
  

 5. REFORMAT determines whether each outn port has a transform function.

•  If an out port does not have a transform function, REFORMAT uses implicit reformat to process the input record. For more information, see “Implicit reformat”.

• If the input record is sent to more than one port, the order of the transform evaluation is sequential: it calls the transform function on each port in order, starting with the lowest numbered port. For example, if the record is to be sent to port0 and port2, it is sent to port0 first, and then to port2. The evaluation of the second transform can depend on the side-effects of the first transform, which means you could make successive calls to a function like next_in_sequence from sequential transforms for the same input record
  

•  If a transform function results in an error or returns NULL, REFORMAT writes the following: 

• An error message to the corresponding error port
• The current input record to the corresponding reject port

•  The component stops execution of the graph when the number of reject events exceeds the reject threshold. 

• if the reject or error ports do not have flows attached to them, REFORMAT discards the record.   

6. REFORMAT writes the record to the out port of each successful transform,and then begins processing the next input record.

 

 

 

 

 

 

Ab Initio Component | REFORMAT: Part 2

 ..Continue from Part 1..

 

output-index(filename or string, optional)

• It specifies either the name of a file containing a transform function, or a transform string. This  component calls the specified transform function for each input record. The transform function should return an index value between 0 and the highest-numbered output port. REFORMAT uses this value to direct the input record to the output port that has the same number as the value, and executes the transform function, if any, associated with that port.

• When you specify a value for this parameter, each input record goes to exactly one transform-output port pair. For example, suppose there are 100 input records and two output ports. Each output port receives between 0 and 100 records. So according to transform function you can specify the output-index split to 50/50, 60/40, 0/100, 99/1 or any other combination which can add up to 100
  

• If an index is out of range (less than zero or greater than the highest-numbered port), the component discards the input record.

    

    NOTE: If you specify a value for the output-index parameter, you cannot also specify the output‑indexes parameter and vice-versa

• If you do not specify a value for either output-index or output-indexes, the component sends every input record to every transform-output port pair.  

 

output-indexes (filename or string, optional)

• It specifies either the name of a file containing a transform function, or a transform string. The component calls the specified transform function for each input record. The transform function uses the value of the input record to direct that input record to particular transform-output ports. 

•  The expected output of the transform function is a vector of numeric values. The component considers each element of this vector as an index into the output transforms and ports. The component directs the input record to the identified output ports and executes the transform functions, if any, associated with those ports. 

• If an index is out of range (less than zero or greater than the highest-numbered port), the component discards the input record.

• If you do not specify a value for either output-indexes or output-index, the component sends every input record to every transform-output port pair.

• As an example of how the component uses the transform function specified in the output-indexes parameter, consider the following:

    out :: output_indexes(in) =
    begin
      out :1: if (in.kind == "x") [vector 0, 1, 2];
      out :2: if (in.kind == "y") [vector 2, 3, 4];
      out : : [vector 5];
    end; 

 

•  When you specify this transform function for the output-indexes parameter, it directs the input record to the transform functions for:

        Ports 0, 1, and 2 if the field in.kind is “x”

        Ports 2, 3, and 4 if the field in.kind is “y”

        Port 5 if otherwise

 logging(boolean, optional)

•     Specifies whether the component logs certain events.

 

 log_input (choice, optional)

• Specifies how often the component sends an input record to its log port. The logging parameter must be set to True for this parameter to be available.

•  For example, if you select 100, the component sends every 100th input record to its log port. 

 

log_output (choice, optional) 

• Specifies how often the component sends an output record to its log port. The logging parameter must be set to True for this parameter to be available.

• For example, if you select 100, the component sends every 100th output record to its log port. 

 

log_reject(choice, optional)

• Specifies how often the component sends a reject record to its log port. The logging parameter must be set to True for this parameter to be available.

• For example, if you select 100, the component sends every 100th reject record to its log port.