The following are the changes in 18.104.22.168 :
New Features :
- Excel export – supports directly saving mined data as an Excel file (details)
- Updated Chromium engine from V54 to V62
Minor changes :
- Default values of ‘Enable Plugins’ and ‘Enable Browser Security’ in Browser Settings set to false (details)
- Browser address bar can be used for Google search
Bug fixes :
- Fixed issues related to handling headers and post data for HTTP requests
- Fixed issue in selecting data using mouse when Zoom-level of browser is not equal to 1 (zoomed in or zoomed out)
- Text formatting issues (line-breaks, spaces) in Capture window fixed
- Fixed issue where order of applying capture-html and capture-more-content was relevant (for applying regex to follow links or to capture images)
- Bug fix in editing keywords. With the previous version changing the first keyword was not possible.
- Minimizes memory usage in mining thread by limiting the number of browser instances created
As always, the latest version may be downloaded and installed from the following page :
This release comes with least bells and whistles since we have not added features or changed cosmetics of the software. But still, this is a major upgrade. The change is all internal.
WebHarvy has been using Microsoft’s Internet Explorer (IE) as its internal browser since inception. Microsoft stopped supporting IE a few years back when they introduced the Edge browser.
So WebHarvy had to switch to another solution to power its internal browser and we believe using Google’s Chrome Browser Project is the way forward. This makes WebHarvy more stable, faster and secure. Switching to Chrome also opens up the possibility of porting the software to other platforms like Mac and Linux.
You may download and install the latest version which is based on Chrome browser from the following link.
As mentioned before the change from IE to Chrome is internal to the software and transparent to the user interface. So, the configuration process and user interface of WebHarvy remains the same.
- For scraping data from sites which require login, the steps have been simplified. You no longer need to login to the website separately from IE. See https://www.webharvy.com/articles/sites-requiring-login.html
- The ‘Internet Options’ menu option under Edit menu has been removed. Instead a new Browser options tab has been added in Settings window.
Running configuration files created using the older version which was based on IE on this new version based on Chrome
Configuration files created using the old version should normally work fine with the new version which is based on Chrome, but there will be exceptions. In such cases we recommend that you create a new configuration using the latest version.
As always, in case you have an questions or need assistance you may contact our support at https://www.webharvy.com/support.html
The main changes in this release are :-
- Increased size of virtual browser used by miner
The dimensions of miner’s virtual browser has been increased. This solves issues related with websites whose layout changes when the browser has a smaller window dimension (mobile layout). This also helps the miner to load more items in a single page and scroll, in case of websites which display data based on the size of the browser window.
- Support for ‘Load more content‘ & ‘Scroll to load next page‘ type pagination even when the real listing page is reached by clicking links/buttons from the start page.
In earlier versions if the listing page loads more data in same page via a button/link click or scroll and if initial navigation (click, java-script etc.) is required in the configuration itself to load the listing page from another start page, then pagination would fail. This release removes this limitation.
- More support for extracting data from popups.
- SQL data export encoding issue related to foreign languages fixed.
Encoding issues while exporting text in non-English languages like Chinese fixed.
- Other minor bug fixes
As always you may download and install the latest version from https://www.webharvy.com/download.html.
Posted in Release update, WebHarvy, WebHarvy Feature
In this blog post, we will take a look at how to scrape images from www.pinterest.com in their full sizes.We follow a two stage extraction process to capture the high-res images from pinterest.com.
In the first extraction stage, we capture the image URLs which are present in the listings page. These URLs actually point to smaller sized images (236 Pixels). Then using any Text Editor, we replace the /236x/ with /564x/ in all the URLs.
For example the URL : https://s-media-cache-ak0.pinimg.com/236x/99/….
is modified to : https://s-media-cache-ak0.pinimg.com/564x/99/….
In the second extraction stage, we use ‘Add URLs‘ method to add the modified URLs and scrape the full sized images ((564 Pixels) from each of these URLs using a single WebHarvy configuration.
This method is displayed in the following video :
- Know more about WebHarvy, the easy to use visual web scraper
- WebHarvy video tutorial series
- Various methods of extracting images and image URLs using WebHarvy
- Download free WebHarvy trial
Have any questions ?
This update addresses problems in installing .NET 4.5 on Windows 7 (and earlier Windows versions where .NET 4.5 is not present) during installation process. Only the installer has been updated in this release and WebHarvy application files are unchanged compared to the just previous version. So in case you are already running 22.214.171.124 you can ignore this version.
You may download and try the latest version from https://www.webharvy.com/download.html. Let us know in case you have any questions.
All WebHarvy application files and installation package are digitally signed (Comodo RSA Code Signing CA) and secured. However in case you get the following Smartscreen warning while trying to install the latest version of WebHarvy, please click the ‘More info‘ link and then click the ‘Run anyway‘ button to proceed with the installation.
The above popup message is displayed because we recently changed our .NET dependency from 3.5 to 4.5, thereby considerably reducing the installation package size, and more importantly the code signing agency of our digital certificate has been changed from GlobalSign to Comodo. So the above warning may appear till the new WebHarvy installer gets enough reputation from Microsoft which will take a few weeks time. In case you have any questions or require assistance please do not hesitate to contact our support.
From this release on wards WebHarvy targets (depends on) .NET 4.5 which comes pre-installed on latest Windows editions. This results in smoother installation process, doing away with .NET 3.5 download and install which was previously required. Targeting .NET 4.5 also helps WebHarvy improve performance and resource usage, and to solve issues related to crashes while trying to extract data from certain websites.
The changes in this release are :-
- Depends on .NET 4.5
- Handles pagination where next page link (next link or ‘show more data’ link) contains a number which varies from page to page
As always you may download and try the latest version from https://www.webharvy.com/download.html. Let us know in case you have any questions.