Data-Adaptive and Data-Resilient Software

Recently, I completed a project that required handling a data source with an inconsistent structure and non-standardized data (commonly referred to as dirty data). Each record contained over 400 fields, but the order of these fields varied unpredictably from one record to the next. The data also suffered from inconsistencies within the fields themselves. For example, some records used abbreviations, while others spelled out terms in full. To complicate things further, the data was accessed through a RESTful API (Representational State Transfer).

The Challenge

Dynamically importing this data directly from the REST API into the target application proved to be problematic. The import script would identify malformed records and skip them entirely, resulting in data loss. While the script was resilient in that it could continue functioning despite encountering errors, it was not adaptive. It lacked the ability to intelligently handle the varying structure of the source data.

In simpler terms: the source data was a mess, and I needed to develop a solution that could intelligently manage it.

The Solution: A Staged ETL Approach

To resolve this issue, I applied a staged approach using the ETL process (Extract, Transform, Load), a common method for dealing with problematic data. Here’s how the ETL process works:

• Extract: Data is pulled from one or more sources (such as databases, files, or APIs) and stored in a temporary location.
• Transform (also known as “Data Scrubbing/Cleaning”): The extracted data is analyzed, cleansed, and standardized. This step resolves inconsistencies and errors, transforming the data into the desired structure for the target system.
• Load: The cleaned and standardized data is then imported into the target system, such as a database or application, for end-user access.

For this project, I implemented a data-adaptive approach, which not only ensured resilience but also allowed the software to intelligently handle and cleanse the dirty source data.

Implementing the Data-Adaptive Approach

The concept is straightforward. First, use the API to retrieve the data records and store them in a temporary intermediary file, without attempting any corrections or cleansing at this stage. This essentially dumps the data into a location where it can be processed using a programming language and tools of choice.

During the Transform phase, the software analyzes each row of data to determine the location of each required data field. In simple terms, this step “finds” the relevant data in each record, even when the structure is inconsistent.

Once the necessary data fields are identified and their locations known, the software can iterate through each row, applying logic to cleanse and standardize the data. Afterward, the cleaned data is written into a new, properly structured file that is consistent and ready for import into the target system.

Enhanced Transformation Logic
During the transformation process, I incorporated some additional features. Based on the presence or absence of certain data in each record, the software dynamically generated new data fields that might have been missing from the source. This approach allowed the system to compensate for incomplete records, further improving data integrity.

Pseudocode for the Solution

Here’s a simplified version of the process in pseudocode:


// Step 1: Retrieve data records from the source system
sourceData = retrieveDataFromSource()

// Step 9: Save cleansed data to the target platform
saveToTargetPlatform(cleansedData)


Explanation:

Step 1: Retrieve the dataset from the source.
Step 2: Map the required fields and their attributes to locate them in the source data.
Step 3: Initialize an array to store the cleansed data.
Step 4: Loop through each row of source data.
Step 5: Identify the required data fields in the current row using the field map.
Step 6: Cleanse and standardize each identified field.
Step 7 (Bonus): Add extra fields based on business logic, dynamically creating new fields if needed.
Step 8: Store the cleansed row of data in the output array.
Step 9: Once all rows are processed, save the cleansed data to the target platform for further use.

Conclusion

By employing a data-adaptive approach, I was able to successfully manage a problematic data source with inconsistent structure and content. This solution not only made the system resilient to errors but also capable of dynamically correcting and adapting to the data it processed. The staged ETL approach, with enhancements during the transformation phase, ensured that the data was accurately cleansed and properly structured for importing into the target application.

The Dead Internet Theory: Separating Fact from Fiction

In recent years, a concept known as the “Dead Internet Theory” has emerged, suggesting that the internet as we know it is slowly dying or becoming obsolete. This theory has sparked debates and raised questions about the future of the digital age. In this article, we will explore the Dead Internet Theory, its origins, its arguments, and the reality behind this provocative concept.

Understanding the Dead Internet Theory:

The Dead Internet Theory posits that the internet is losing its original spirit of openness, freedom, and decentralization. Proponents argue that increasing corporate control, government surveillance, censorship, and the dominance of a few powerful entities are stifling innovation and transforming the internet into a highly regulated and centralized platform. They claim that the internet is losing its vitality and becoming a mere reflection of offline power structures.

Origins and Key Arguments:

The origins of the Dead Internet Theory can be traced back to concerns raised by early internet pioneers and activists who championed a free and open internet. They feared that the commercialization and consolidation of online platforms would undermine the principles that made the internet a transformative force.

Among the concerns put forth by the Dead Internet Theory are fake traffic from bots, and fake user accounts. According to numerous sources, a significant percent of Internet traffic is from bots. Estimates range from a low of 42% of traffic, to over 66% of traffic is from bots. Bots are automated software programs that perform various tasks, ranging from simple to complex, on the internet. They can be beneficial, such as search engine crawlers or chatbots, or malicious, like spam bots or distributed denial-of-service (DDoS) attack bots.

The problems associated with bot traffic arise primarily from malicious bots that engage in fraudulent activities, spamming, data scraping, click fraud, credential stuffing, and more. These activities can have severe consequences, including financial losses, compromised security, reputational damage, and disruptions to legitimate online services.

Social media platforms have been grappling with the challenge of fake user accounts for some time. These accounts are created for various purposes, including spreading misinformation, engaging in spam or fraudulent activities, manipulating public opinion, or conducting malicious campaigns. Increasingly, bots use these fake accounts “push” a narrative. This activity is very noticeable during political elections.

Proponents of The Dead Internet Theory highlight several key arguments:

• Centralization and Monopolistic Power: They argue that a small number of tech giants now dominate the internet, controlling vast amounts of data and shaping user experiences. This concentration of power limits competition and stifles smaller players’ ability to innovate.
• Surveillance and Privacy Concerns: With the rise of surveillance technologies and data breaches, privacy advocates express worry that individuals’ online activities are constantly monitored and exploited for various purposes, eroding trust in the internet.
• Censorship and Content Control: The theory also highlights instances of government-imposed censorship, content moderation challenges, and algorithmic biases, suggesting that freedom of expression is under threat.
• Net Neutrality and Access: Advocates argue that the internet’s openness is compromised by practices that prioritize certain types of traffic or restrict access based on geographic location or socioeconomic factors, leading to a digital divide.

The Reality:

While the concerns raised by the Dead Internet Theory hold some validity, it is important to approach the subject with nuance. The internet remains a dynamic and evolving medium, shaped by technological advancements and societal changes. While challenges exist, numerous initiatives and movements aim to preserve the internet’s founding principles.

Efforts such as decentralized technologies (like blockchain), open-source software, encryption tools, and net neutrality advocacy strive to counteract centralization, surveillance, and censorship. Additionally, the proliferation of alternative platforms, social networks, and online communities ensures that diverse voices and opinions can still find a space online.

Final Thoughts:

The Dead Internet Theory serves as a reminder of the ongoing struggle to maintain an open, free, and decentralized internet. While concerns over centralization, surveillance, and censorship are valid, the internet is not irreversibly “dead.” It continues to evolve, driven by the collective actions of individuals, organizations, and policymakers. It is important that the internet remains a powerful tool for connectivity, knowledge-sharing, and empowerment.

Targeted Marketing Models

Targeting the best prospective core customers within the markets that have the highest probability of success establishes clear marketing goals that can be accurately measured by customer acquisition and sales conversion.

In a previous post I covered Marketing Uplift & Predictive Modeling.

This post will discuss using Predictive Modeling & Marketing Uplift combined with classic Customer Analytics to create more accurate Targeted Marketing Campaigns.

The foundation of any accurate marketing model is a precise definition of your core customer, which encompasses 4 key aspects:

1. Who your best customers are
2. Where your best potential customers can be found
3. What messages those customers respond to & when to send them
4. The value potential the customers have to you, either in terms of dollars or visits

Customer analytic solutions have traditionally been used to gather some of those customer insights(Data Dimensions).

However, a properly defined Marketing Model combined with Customer Analytics can help not only accurately quantify the benefit for your marketing programs, but build out a deeper definition of your core customer.

Predictive Modeling can be used to understand the potential benefit of a new marketing program before it is launched. It can help you predict if it is a good investment or a bad one.

It can also be used to optimize existing marketing programs by establishing and analyzing key metrics.

The ability to measure, analyze, and assess existing marketing programs can identify which programs are not meeting their potential and then determine the gap between actual performance and ultimate potential.

Correlating this marketing information with known customer data dimensions allows you to refine your Core Customer Profile.

This insight allows you to immediately reprioritize resources for management, re-assign budget and marketing from the initiatives with no upside to the ones that are just underperforming.

Insights from customer analytics and marketing models on who core customers are, where potential core customers are located within an underperforming market area, what messages they respond to and when they respond to them, and the potential value of each of those core customers makes it possible to develop a focused demographic & psychographic profile of customers to target, and a measurable marketing campaign to go after them.

BriteWire sends the right message, to the right customer, at the right time.

Marketing Uplift & Predictive Modeling

Marketing Uplift(aka Marketing Lift) and Predictive Modeling are hot concepts in marketing. In this post we take a quick look at these marketing techniques.

Marketing Uplift(aka Marketing Lift) is the difference in response rate between a treated group and a randomized control group.

Marketing Uplift Modeling can be defined as improving (upping) lift through predictive modeling.

A Predictive Model predicts the incremental response to the marketing action. It is a data mining technique that can be applied to engagement and conversion rates.

Uplift Modeling uses both the treated and control customers to build a predictive model that focuses on the incremental response.

Traditional Response Modeling only uses the treated group to build a predictive modeling. Predictive Modeling separates the likely responders from the non-responders.

Traditional Response Modeling segments an audience into the following primary groups:

• The Persuadables : audience members who only respond to the marketing action because they were targeted
• The Sure Things : audience members who would have responded whether they were targeted or not
• The Lost Causes : audience members who will not respond irrespective of whether or not they are targeted
• The Do Not Disturbs or Sleeping Dogs : audience members who are less likely to respond because they were targeted

The only segment that provides true incremental responses is the Persuadables.

Because uplift modelling focuses on incremental responses only, it provides very strong return on investment cases when applied to traditional demand generation and retention activities. For example, by only targeting the persuadable customers in an outbound marketing campaign, the contact costs and hence the return per unit spend can be dramatically improved.

One of the most effective uses of uplift modelling is in the removal of negative effects from retention campaigns. Both in the telecommunications and financial services industries often retention campaigns can trigger customers to cancel a contract or policy. Uplift modelling allows these customers, the Do Not Disturbs, to be removed from the campaign.

Big Data & Psychometric Marketing

Psychometrics, sometimes also called psychographics, focuses on measuring psychological traits, such as personality. Psychologists developed a model that sought to assess human beings based on five personality traits, known as the “Big Five.” Big data correlated with personality profiles allows for accurate psychographic targeting by marketers.

Psychometrics is a field of study concerned with the theory and technique of psychological measurement. Psychometric research involves two major tasks. 1. The construction of instruments. 2. The development of procedures for measurement. Practitioners are described as psychometricians. The most common model for expressing an individual’s psychometric personality are the Big Five personality traits.

5 Personality Traits (Big Five)

• Openness (how open you are to new experiences?)
• Conscientiousness (how much of a perfectionist are you?)
• Extroversion (how sociable are you?)
• Agreeableness (how considerate and cooperative you are?)
• Neuroticism (are you easily upset?)

Based on these 5 dimensions, also known as OCEAN(Openness, Conscientiousness, Extroversion, Agreeableness, Neuroticism) we can make a relatively accurate assessment of a person. This includes their needs and fears, and how they are likely to behave.

The “Big Five” has become the standard technique of psychometrics. The problem with this approach has been data collection, because it required filling out a lengthy, highly personal questionnaire.

With the Internet, and cell phones this is no longer a problem.

The Internet allows researchers to collect all sorts of online data from users and using Data Correlation they can associate online actions with personality types. Remarkably reliable deductions can be drawn from simple online actions. What users “liked,” shared or posted on Facebook, or what gender, age, place of residence they specified can be directly correlated with personality traits.

While each individual piece of such information is too weak to produce a reliable prediction, when tens, hundreds, or thousands of individual data points are combined the resulting predictions become very accurate. This enables data scientists and marketers to connect the dots and make predictions on people’s demographic and psychographic behavior with astonishing accuracy.

Smartphones are a psychological questionnaire on their users that is constantly being updated, both consciously and unconsciously. It is not uncommon for data analytics companies to have over 4,000 data points for each person in their target data set.

The strength of their psychometric modeling was illustrated by how well it could predict a subject’s answers.

It is now possible to assign Big Five values based purely on how many profile pictures a person has on Facebook, or how many contacts they have (a good indicator of extroversion).

Psychological profiles be created from this data, but the data can also be used the other way round… to search for specific profiles.

This allows for accurate psychographic targeting by marketers in ways that have never before been possible.

BriteWire Intelligent Internet Marketing facilitates building Psychometric User Profiles with purposefully designed content interactions, and automates marketing responses based on those triggers.

PropTech Disrupting The Real Estate Industry

Real Estate is the largest and most valuable asset class in the world but the Real Estate industry is ripe for disruption. Traditional real estate brokerages are under increasing competitive pressure as technology companies make inroads into the previously walled off real estate market. This complex, multi-faceted and high-stakes industry is rapidly becoming one of the hottest markets for entrepreneurs and investors alike.

PropTech(short for Property Technology) refers to the sector of startups and new technologies cropping up in response to decades of inefficiencies and antiquated processes in the real estate industry. The term is being used to encapsulate the entire market space of technology and real estate coming together to disrupt the traditional real estate model. PropTech is also referred to as CREtech (Commercial Real Estate Technology) or REtech (Real Estate Technology).

The PropTech sector is generating increased buzz as more people realize the opportunities for innovation in this sector, and venture capital investments expand. Real estate tech startups around the globe raised $1.7 billion worth of investments in 2015 – that’s an 821% increase in funding compared to 2011’s total. PropTech companies have raised around $6.4 billion in funding from 2012 to 2016. It is difficult to pin down exact numbers, but it is estimated that investment in the space increased by 40% for 2016. Compass, Homelink, SMS Assist and OpenDoor Labs all saw their valuations increase to over $1 billion in 2016.

Internet technology has tremendous potential to improve transparency and efficiency in real estate transactions. Property Listings and Property Search was the initial area for improvement. Over the past several years, technology advancements in property search and listing engines have been introduced.

These advancements allow home buyers to more easily find their home by making search criteria easier to specify. Buyers can specify the obvious search criteria like price range, size, number of bedrooms, and location. Additional facets of navigation like lot size, water features, adjacent public land, conservation easements, and if horses are allowed can also be used as search criteria. Buyers can view maps of where the homes for sale are located, and find homes for sale using maps as their primary search navigation tool. Search criteria can be saved in property search engines so that when new homes come on the market that match a buyer’s predefined criteria they are automatically notified.

Advancements on property listings have allowed a listing to easily be syndicated to hundreds of websites with the click of a mouse. This dramatically expands the listing’s exposure to potential buyers.

Technological advancements in deep learning, AI and other big data technologies are driving significant innovation in all areas of the property technology sector. Advancements in video, 3D Modeling, and virtual reality are allowing buyers to virtually tour homes they are interested in. These advancements allow agents to more effectively showcase the properties they are selling. However, these advancements in moving key data to the Internet may also mitigate the role that real estate agents play in the real estate process.

Competition in the space is increasing, but many tech companies do not have access to the industry and associated data due to real estate laws and regulations on disclosure. In many states it is illegal to disclose the price of a real estate transaction, and only members of the local real estate MLS have access to all the listing information.

What is next? Investor confidence is high and reports are that a massive amount of investment capital will be pumped into the real estate sector. It is safe to assume that we will see significant changes in the technical landscape for the real estate market.