This article aims to test esProc’s performance in processing text files using an example of data query and filtering and through the comparison with Java and Perl performing the same processing.
The orders.txt file storing order records is used as the test data. It is imported in esProc as follows:
ORDERID CLIENT SELLE...
2016-05-27 4545 0
1
During data computing in esProc, raw data, intermediate data, and the final resulting data consume certain system resources. That includes the memory consumption, database connection usage and the space occupation by the file objects. Too much resource consumption may cause memory overflow, database access failure and ...
2016-05-20 6094 4
2
1.Foreign key reference
The foreign key is a common concept for various relational databases. For a relational table, you can define one or multiple fields as the foreign key, through which an association with the data in another table can be created. For example, in a game scoring report, often players are recorded wi...
2016-05-17 2614 0
1
With databases, creating appropriate indexes for tables can greatly increase query efficiency. Similarly, you can create index sequences for record sequences or table sequences in esProc to enhance the efficiency of querying data repeatedly.
For example, you need to query the food order file Order_Foods.txt repeatedly....
2016-05-09 2467 0
0
Numerical computations in data reporting are often done with some degree of accuracy by rounding off numerical values to their nearest numbers, which is one type of the rounding operations. Yet this can result in data inconsistency. And in that case, the rounded numbers need to be adjusted to restore the consistency. T...
2016-04-29 2631 0
1
1. Testing purposes
Test esProc and Oracle on the same hardware to compare their single machine performances in carrying out a single big task and small concurrent tasks.
2. Testing contents and methods
Data volume:
Small data volume: Single fact table around 10G. To prevent the testing result from being affected by th...
2016-04-25 2052 0
1
Date and time data has its own characteristics for analysis and handling. In this article let’s look at how to perform date and time handling in esProc.
1. Conversion to date and time data
Usually the date and time data is entered or displayed as strings. With esProc, you can click Tool>Option to set the default for...
2016-04-21 2317 0
0
It’s common to handle structured text files in Java programming. But since Java lacks corresponding class library, code produced through hardcoding is complicated and difficult to maintain.
For example, sales.txt is a tab-separated structured text file that needs sorting by SellerId. Here’s the code:
Comparator<...
2016-04-18 2053 0
0
An esProc set is an ordered set. Therefore, its members can be referenced by sequence numbers. A flexible exploitation of sequence numbers will give full play to esProc’s capability, handling a computation in a simpler and more efficient way.
1. Accessing members
Therefore it is recommended that certain esProc function...
2016-04-15 2003 0
0
esProc uses various variables, including the cell variable, in all its operations. Each type of variable has its scope, which is the topic the article discusses.
1. esProc Parameters and Variables
esProc allows defining the globally-used data as cellset constant, like the following sequence-type constant weekdays:
The...
2016-04-13 2080 0
0
esProc applies set theory in a uniquely deep, common way, compared with traditional programming languages. The sequence in esProc is in essence a type of set. It’s important to learn to think in sets all the time when working with esProc.
1.Sequences and sets in esProc
The sequence, as well as integer and string, is on...
2016-04-11 2042 0
0
In data analysis, comparison operations are employed to check whether a datum is greater/less or equal to another datum, as well as to perform operations including query, select, sort and group. Here we discuss the uses of comparison operations in esProc and solve possible related problems.
1.Comparing single values
es...
2016-04-08 1898 0
0
Programmers can handle structured data using SQL statements when the data is stored in a single database. But they can’t use SQL statements directly when the data is held in a non-database such as the file, the NoSQL database, or the Json data stream, or when multiple databases are involved. As an alternative, programm...
2016-04-06 2085 0
1
esProc provides regex() function to match a string or a string field value in a record sequence using a pattern defined by the regular expression. The aim is to analyze and examine the string in order to find the pattern in it and replace it. Here we look at the uses for regular expression in esProc.
1. The basics
A re...
2016-04-01 1873 0
0
To deal with a large text file that can’t be wholly held in the memory, you need to import it in segments and process each segment separately. It’s knotty. Sometimes even multithreaded parallel processing is needed so as to increase performance. But since most of the programming languages don’t support basic class libr...
2016-03-23 1607 0
0
